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Information  Technology  Laboratory.  The  IUSR  goals  are  to: 


• Encourage  software  supplier  and  purchaser  organizations  to  work  together  to  understand  user  needs 
and  tasks; 
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formative  testing,  testing  with  hardware. 
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Introduction 

The  Common  Industry  Specification  for  Usability  - Requirements  (CISU-R)  helps  usability  professionals, 
product  managers,  and  others  working  in  product  design  and  development  to  create  usability  requirements.  It 
sets  standards  for  specifying  usability  requirements,  which  include  three  types  of  information: 

- The  context  of  use:  the  intended  users,  their  goals  and  tasks,  associated  equipment,  and  the 
physical  and  social  environment  in  which  the  product  can  be  used. 

- Performance  and  satisfaction  criteria:  measures  of  usability  for  the  product. 

- The  test  method  and  context  of  testing:  the  method  to  be  used  to  test  whether  the  usability 
requirements  have  been  met  and  the  context  in  which  the  measurements  will  be  made. 

Underlying  the  specific  goals  of  the  standard  is  a deeper  goal  of  creating  useful  and  usable  products  that  allow 
users  to  complete  their  tasks  efficiently,  effectively,  and  with  satisfaction.  To  support  this  deeper  goal,  the 
CISU-R  provides  a structure  for  requirements  to  help  stakeholders: 

- Document  the  context  of  use  for  a product,  including  definitions  of  the  expected  technical,  physical, 
and  social  environments,  user  groups,  goals  for  use  of  the  product  and  scenarios  of  use,  as  defined  in 
Clause  6.2. 

- Write  usability  requirements  in  sufficient  detail  to  make  an  effective  contribution  to  design  and 
development.  (A  suggested  outline  for  usability  requirements  is  provided  in  Annex  A.) 

- Relate  usability  requirements  to  stakeholder  requirements  (including  user,  customer,  and  business) 
for  successful  use  of  a product  and  increased  productivity. 

- Define  usability  criteria  that  can  be  empirically  validated.  (Annex  B provides  examples  of  usability 
criteria.) 

- Define  the  method  for  testing  the  product  against  the  criteria.  (Annex  C identifies  the  information  to  be 
specified  for  the  test  method.) 

- Create  requirements  that  are  useful  throughout  the  product  design  and  development  process, 
providing  input  to  the  design  process  early  in  a project  and  adding  more  detailed  information  about 
criteria  and  methods  as  it  is  available. 

Usability  requirements  created  with  the  CISU-R  can  be  updated  throughout  a product's  lifecycle.  A project 
may  choose  to  create  complete  requirements,  including  all  information  specified  in  the  CISU-R,  or  may  adopt 
a level  of  conformance  that  meets  stakeholder  and  business  goals. 

The  CISU-R  can  help  integrate  the  creation  of  usability  requirements  into  any  design  or  development  process. 
(Annex  D contains  an  example  of  a user  centered  design  process  that  incorporates  the  development  of 
usability  requirements  under  the  CISU-R.) 

- Requirements  can  meet  one  of  three  levels  of  compliance  with  the  CISU-R.  Each  level  builds  on  the 
previous  one,  allowing  the  usability  requirements  to  be  developed  over  time,  with  increasing  detail  and 
precision.  This  approach  allows  the  CISU-R  to  be  used  for  all  types  of  projects,  from  the  smallest  or 
most  informal  to  complex  or  formally  specified  products. 


Customers,  users,  and  development  teams  can  use  the  CISU-R  as  a communications  tool  to 
understand  and  specify  requirements  within  an  organization  or  as  the  basis  for  a contractual 
relationship  between  companies.  The  three  levels  of  compliance  provide  the  flexibility  to  match  the 
level  of  detail  and  formality  of  the  usability  requirements  to  business  needs. 
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- Requirements  may  also  include  indirect  measures  of  usability,  such  as  post-release  satisfaction  tests, 
or  business  performance  requirements,  such  as  completed  transactions.  These  criteria  are  not 
intended  for  evaluation  using  a summative  usability  test,  as  required  by  the  third  level  of  conformance, 
but  can  be  useful  in  establishing  business  or  performance  requirements. 

- The  CISU-R  recognizes  that  usability  requirements  are  just  one  type  of  product  requirement.  They 
complement  functional,  business,  process,  safety,  and  non-functional  (quality)  requirements.  The 
information  gathered  in  creating  usability  requirements  can  be  used  to  help  define  other  user 
requirements  (for  example,  for  features  of  the  interface).  (Annex  E lists  other  types  of  user  and 
usability  requirements.) 

The  CISU-R  focuses  on  the  information  needed  to  create  usability  requirements  that  are  useful  for  design  and 
development,  rather  than  dictating  a specific  process  or  user  centered  design  activities.  (Annex  E contains  an 
example  of  a user  centered  design  process  that  incorporates  the  CISU-R.) 

- In  situations  where  there  are  already  processes  and  document  formats  in  place,  stakeholders  can 
incorporate  the  CISU-R  into  that  process  by  ensuring  that  the  information  specified  here  is  included  in 
any  usability  requirements. 

- If  there  is  no  established  format,  requirements  can  use  the  information  specified  in  Annex  A as  a 
guide.  (Examples  of  all  three  levels  are  included  in  Annexes  F,  G,  and  H as  a reference.) 

The  process  of  creating  requirements  with  the  CISU-R  can  be  incorporated  into  many  user  centred  design 
processes.  For  example,  in  ISO  13407,  specifying  requirements  is  one  activity  in  an  iterative  process.  The 
CISU-R  provides  details  for  how  to  complete  this  activity,  as  shown  in  Figure  1 . 
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Figure  1 — Requirements  in  the  User-Centered  Design  Process 


To  ensure  that  the  requirements  are  correct,  other  user  centered  design  activities  (such  as  competitive 
analysis,  interviews,  surveys,  focus  groups,  field  studies,  task  analysis,  benchmark  usability  tests,  or  paper 
prototyping)  can  be  used  early  in  the  development  process  to  obtain  feedback  from  users  to  iteratively  refine 
requirements. 

The  CISU-R  complements  other  user  centered  design  standards. 

- It  uses  the  definition  of  usability  in  ISO  9241-1 1 : the  effectiveness,  efficiency,  and  satisfaction  with 
which  the  intended  users  can  achieve  their  tasks  in  the  intended  context  of  product  use. 
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- Usability  requirements  created  using  the  CISU-R  are  consistent  with  the  recommendations  of  ISO 
13407. 

- Usability  tests  conducted  to  measure  whether  usability  requirements  have  been  met  can  be  reported 
using  the  Common  Industry  Format  (CIF)  (ISO/IEC  25062). 

- The  development  of  usability  requirements  may  be  supported  by  the  use  of  standards  such  as  ISO 
9241  or  ISO  9126. 
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1.  Scope 

1.1  General 

The  CISU-R  describes  how  to  specify  usability  requirements  for  both  hardware  and  software  products.  It  is 
consistent  with  the  definition  of  usability  in  ISO  9241-11:  effectiveness,  efficiency,  and  satisfaction  in  a 
specified  context  of  use. 

Usability  requirements  based  on  the  CISU-R  have  three  parts: 

- The  context  of  use:  intended  users,  their  goals  and  tasks,  associated  equipment,  and  the  physical 
and  social  environment  in  which  the  product  will  be  used,  and  examples  of  scenarios  of  use. 

- Performance  and  satisfaction  criteria:  measures  of  usability  for  the  product. 

- The  test  method  and  context  for  testing:  the  means  of  determining  whether  the  usability 
requirements  have  been  met. 

These  requirements  can  be  specified  to  different  levels  of  detail,  to  meet  three  levels  of  conformance  to  this 
standard,  at  different  stages  of  a product  design  and  development  process. 

The  CISU-R  only  applies  to  usability  requirements,  one  of  many  types  of  requirements  for  hardware  and 
software  products.  Other  types  of  requirements  include  non-usability  functional  requirements,  process 
requirement,  business  requirements,  safety  requirements,  and  non-functional  (quality)  requirements. 

Usability  requirements  specified  with  the  CISU-R  may  be  used  alone,  or  may  be  incorporated  into  a broader 
requirements  gathering  process  or  other  software  or  systems  development  process. 

The  CISU-R  does  not  prescribe  any  specific  development  process.  The  CISU-R  does  not  prescribe  specific 
measures  for  usability. The  CISU-R  does  not  prescribe  any  specific  user  centered  design  process,  but  it  is 
consistent  with  the  relevant  process  in  ISO  13407. 

1.2  Relationship  to  other  standards 

1.2.1.  General 

The  CISU-R  can  be  used  in  conjunction  with  other  related  standards. 

1 .2.2.  Relationship  to  ISO/IEC  25062  - Common  industry  format  for  usability  test  reports 

As  a tool  to  support  testing  and  verification,  the  requirements  specified  in  the  CISU-R  can  be  tested  and  the 
results  reported  using  the  ISO/IEC  25062  (Common  Industry  Format  (CIF)  for  usability  test  reports). 
Organizations  can  develop  their  test  and  verification  plans  directly  from  the  CISU-R. 
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1.2.3.  Relationship  to  ISO  13407  - Human  centred  design  processes  for  interactive  systems 

ISO  13407  defines  one  of  the  stages  of  user  centered  design  as  the  specification  of  requirements.  Usability 
requirements  created  using  the  CISU-R  are  consistent  with  the  recommendations  of  ISO  13407. 

The  CISU-R  directly  supports  the  following  objectives  in  ISO  13407:1999  7.3.2: 

“The  specification  of  user  and  organizational  requirements  should 

a)  identify  the  range  of  relevant  users  and  other  personnel  in  the  design, 

b)  provide  a clear  statement  of  the  human-centered  design  goals, 

c)  set  appropriate  priorities  for  the  different  requirements, 

d)  provide  measurable  criteria  against  which  the  emerging  design  can  be  tested, 
g)  be  adequately  documented.” 

1.2.4.  Relationship  to  ISO/IEC  9126  - Software  Engineering  - Product  Quality 

Usability  requirements,  as  defined  in  the  CISU-R,  are  used  in  conjunction  with  other  detailed  design 
requirements,  such  as  those  in  Annex  B.  The  development  of  these  detailed  requirements  may  be  supported 
by  the  use  of  standards  such  as  ISO  9241  or  ISO/IEC  9126 

The  CISU-R  enables  usability  to  be  specified  as  an  outcome  of  interaction,  measured  as  user  performance 
and  satisfaction.  ISO/IEC  9126-1  defines  this  as  a high  level  quality  objective:  the  user's  experience  of  the 
“quality  in  use",  which  is  determined  not  only  by  the  features  of  the  user  interface,  but  also  by  the  extent  to 
which  the  functionality  supports  the  user’s  tasks  and  by  non-functional  requirements  such  as  acceptable 
computer  performance  and  reliability. 

Examples  of  metrics  for  usability  requirements  can  be  found  in  ISO/IEC  9126  parts  2 and  3. 

1.2.5.  Relationship  to  ISO/IEC  25030  - Quality  requirements  and  guide 

Usability  requirements  in  this  document  correspond  to  the  quality  in  use  requirements  in  ISO/IEC  25030. 


2.  Conformance 

2.1  General 

A usability  requirements  specification  conforming  to  the  CISU-R  may  either  consist  of  information  incorporated 
into  existing  documents  within  an  organization  or  may  be  produced  as  a stand-alone  document,  which  may 
use  the  format  in  Annex  A. 

A usability  requirements  specification  conforms  to  the  CISU-R  if  it  meets  the  requirements  (stated  as  "shall”) 
in  Clause  6 for  one  of  three  levels  of  compliance.  The  recommendations  (stated  as  "should”)  should  be 
implemented  whenever  appropriate. 

The  three  levels  of  compliance  also  allow  conforming  requirements  to  be  written  at  an  appropriate  level  of 
detail  for  each  product  and  development  process.  Each  of  these  levels  builds  on  the  previous  level(s), 
allowing  requirements  to  be  specified  iteratively,  adding  additional  information  as  the  requirements  reach  the 
next  level  of  compliance,  as  shown  in  Figure  2. 

The  three  levels  also  allow  the  requirements  to  be  specified  iteratively,  adding  more  detailed  information  about 
criteria  and  methods  as  it  is  available,  and  providing  useful  input  throughout  the  product  design  and 
development  process.  Each  level  can  be  completed  at  an  appropriate  stage  or  checkpoint. 
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EXAMPLE  1 In  a process  based  on  ISO13407,  requirements  might  be  developed  to  Level  1 during  the  “Understand 
Context  of  Use”  activity;  to  Level  2 during  the  “Specify  Requirements”  activity;  and  to  Level  3 during  the  “Evaluate 
Designs”  activity. 

EXAMPLE  2 Requirements  might  be  developed  to  Level  1 during  work  on  conceptual  prototypes;  to  Level  2 as  the 
user  interface  design  is  specified;  and  to  Level  3 for  summative  testing  of  the  product. 


LEVEL  1 LEVEL  2 

LEVEL  3 

Specify  the  context  of  use 

Identify  usability  Specify  criteria  and 

criteria  identify  target  values 

Identify  specific 
values 

Identify  possible  Specify  evaluation 

evaluation  methods  methods 

Specify  usability  test 
protocol 

Figure  2:  Information  included  in  requirements  at  each  level  of  compliance 


Any  statement  of  conformance  shall  indicate  which  level  of  compliance  has  been  achieved. 
EXAMPLE  This  document  conforms  to  Level  X of  the  CISU-R. 


2.2  Level  1 Compliance 

Requirements  meeting  Level  1 compliance  include  a description  of  the  context  of  use  and  identify  both  the 
types  of  criteria  and  methods  to  evaluate  them  that  can  be  used  to  measure  a product’s  success  in  meeting 
the  requirements.  At  Level  1,  evaluation  methods  are  identified  as  an  input  to  planning,  and  will  be  specified  in 
more  detail  in  later  levels. 

For  Level  1 compliance,  usability  requirements  shall: 

- Specify  all  of  the  information  describing  the  context  of  use  (see  Clause  6.2). 

- Identify  relevant  criteria  for  use  of  the  product  (see  Clause  6.3. 1 ). 

- Identify  appropriate  evaluation  methods  (see  Clause  6.4. 1 ). 

2.3  Level  2 Compliance 

Requirements  meeting  Level  2 compliance  add  more  detailed  information  about  performance  and  satisfaction 
criteria,  specifying  the  criteria  to  be  used  and  identifying  target  values  (or  a method  for  determining  those 
values).  At  this  level,  requirements  also  specify  the  evaluation  methods  including  a description  of  how  the  test 
will  be  conducted. 
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For  Level  2 compliance,  usability  requirements  shall: 

- Specify  all  of  the  information  describing  the  context  of  use  (see  Clause  6.2). 

- Specify  evaluation  criteria,  and  target  values  (see  Clause  6.3.2). 

- Specify  a usability  evaluation  method,  and  identify  the  context  for  evaluation  (see  Clause  6.4.2). 

2.4  Level  3 Compliance 

For  Level  3 compliance,  requirements  include  a specification  of  target  values,  that  have  been  validated,  for  the 
performance  and  satisfaction  criteria,  and  a complete  protocol  for  the  usability  test  method. 

NOTE  1 For  some  products,  usability  requirements  may  not  reach  Level  3 compliance  until  close  to  the  end  of  the 
design  and  development  process;  for  others,  Level  3 compliance  may  be  part  of  a contractual  agreement. 

NOTE  2 The  usability  test  specified  in  Level  3 may  be  reported  using  ISO/IEC  25062  (Common  Industry  Format  (CIF) 
for  usability  test  reports).  If  use  of  this  report  format  is  planned,  the  protocol  should  include  a way  to  meet  all  requirements 
of  this  standard. 

For  Level  3 compliance,  usability  requirements  shall: 

- Specify  all  of  the  information  describing  the  context  of  use  (see  Clause  6.2). 

- Specify  target  values  for  each  criterion  and  user  group  (see  Clause  6.3.3). 

- Specify  a complete  protocol  for  the  usability  test  method,  as  specified  in  Annex  C (see  Clause  6.3.3). 

3.  Normative  references 

The  following  referenced  documents  are  indispensable  for  the  application  of  this  document.  For  dated 
references,  only  the  edition  cited  applies.  For  undated  references,  the  latest  edition  of  the  referenced 
document  (including  any  amendments)  applies. 

ISO/IEC  DTR  9126-2  (2001):  Software  Engineering  - Product  quality  - Part  2:  External  metrics 

ISO/IEC  DTR  9126-3  (2001):  Software  engineering  - Product  quality  - Part  3:  Internal  metrics 

ISO/IEC  DTR  9126-4  (2001 ):  Software  engineering  - Product  quality  - Part  4:  Quality  in  use  metrics 

ISO  9241-11  (1998):  Ergonomic  requirements  for  office  work  with  visual  display  terminals  (VDTs)  - Guidance 
on  usability 

ISO  13407  (1999)  Human  centred  design  processes  for  interactive  systems 

ISO/IEC  CD  25030:  Software  Product  Quality  Requirements  and  Evaluation  (SQuaRE)  - Quality  requirements 
and  guide 

ISO/IEC  DIS  25062:  Software  engineering  — Software  product  Quality  Requirements  and  Evaluation 
(SQuaRE)  — Common  Industry  Format  for  usability  test  reports 

4.  Terms  and  definitions 

This  document  uses  the  following  definitions. 
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4.1 

usability 

the  extent  to  which  a product  can  be  used  by  specified  users  to  achieve  specified  goals  with  effectiveness, 
efficiency,  and  satisfaction  in  a specified  context  of  use 
[ISO  9241-11:1998] 

4.2 

effectiveness 

the  accuracy  and  completeness  with  which  users  achieve  specified  goals 
[ISO  9241-11:1998] 

4.3 

efficiency 

the  resources  expended  in  relation  to  the  accuracy  and  completeness  with  which  users  achieve  goals 
[ISO  9241-11:1998] 

4.4 

requirements 

expression  of  a perceived  need  that  something  be  accomplished  or  realized 
[ISO/IEC  FCD  25030] 

NOTE  The  requirements  may  be  specified  as  part  of  a contract,  or  specified  by  the  development  organization,  as 
when  a product  is  developed  for  unspecified  users,  such  as  consumer  software,  or  the  requirements  may  be  more 
general,  as  when  a user  evaluates  products  for  comparison  and  selection  purpose. 


4.5 

satisfaction 

freedom  from  discomfort,  and  positive  attitudes  towards  the  use  of  the  product 
[ISO  9241-11:1998] 

4.6 

context  of  use 

the  users,  tasks,  equipment  (hardware,  software,  and  materials),  and  the  physical  and  social  environments  in 
which  a product  is  used 
[ISO  9241-11:1998] 

NOTE  1 The  physical  environment  includes  the  workplace  design  (furniture,  posture,  and  location)  and  the  visual, 
auditory,  thermal,  and  atmospheric  conditions,  as  well  as  any  health  and  safety  hazards. 

NOTE  2 The  social  environment  includes  social  and  organizational  issues  including  the  work  organization  and  structure, 
availability  of  assistance,  and  interruptions,  presence  of  other  people,  job  design,  and  autonomy. 

4.7 

measure  (noun) 

variable  to  which  a value  is  assigned  as  the  result  of  measurement 
[ISO/IEC  15939:2002] 

NOTE  The  term  “measures"  is  used  to  refer  collectively  to  base  measures,  derived  measures,  and  indicators. 

4.8 

measure  (verb) 
make  a measurement 
[ISO/IEC  14598-1:1999] 

4.9 

measurement 

set  of  operations  having  the  object  of  determining  a value  of  a measure 

[ISO/IEC  15939:2002,  based  on  the  definition  in  International  Vocabulary  of  Basic  and  General  Terms  in 
Metrology,  1993] 
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4.10 
user 

a person  who  interacts  with  the  product. 

[ISO  9241-11:1998] 

NOTE  In  this  standard  users  may  include  current  users,  potential  users,  users  with  disabilities,  expected  future  users, 
users  of  the  task  output,  and  staff  who  support  or  maintain  the  product. 

4.11 

user  group 

subset  of  intended  users  that  are  differentiated  from  other  intended  users  by  factors  that  are  likely  to  influence 
usability,  such  as  age,  culture,  knowledge,  skill,  expertise,  role  or  responsibility. 

4.12 

scenarios  of  use 

descriptions  of  how  users  carry  out  their  tasks  in  a specified  context. 

4.13 

stakeholder 

a party  having  a right,  share  or  claim  in  a system  or  in  its  possession  of  characteristics  that  meet  that  party’s 
needs  and  expectations. 

[ISO  15288:2002] 

NOTE  Stakeholders  as  used  in  this  standard  includes  users  and  user  groups 

4.14 
goal 

an  intended  outcome 
[ISO  9241-11:1998] 

4.15 
task 

the  activities  required  to  achieve  a goal 
[ISO  9241-11:1998] 

NOTE  1 These  activities  can  be  physical  or  cognitive. 

NOTE  2 Job  responsibilities  can  determine  goals  and  tasks. 


5.  Purpose 
5.1  General 

The  purpose  of  the  Common  Industry  Specification  for  Usability  - Requirements  (CISU-R)  is  to  provide  a 
structure  and  format  for  defining  usability  requirements,  which  can  be  incorporated  into  any  user  centered 
design  process.  Usability  requirements  specify  the  effectiveness,  efficiency,  and  satisfaction  with  which  the 
intended  users  can  achieve  their  tasks  in  the  intended  context  of  product  use. 

Usability  requirements  developed  with  the  CISU-R  can  be  incorporated  into  more  comprehensive 
requirements  documents,  or  can  be  created  as  a stand-alone  document.  In  either  case,  to  make  an  effective 
contribution  to  design  and  development,  the  requirements  can: 

- Be  based  on  accurate  information  about  actual  or  potential  users  and  their  tasks 

- Provide  sufficient  detail  to  contribute  to  other  user-centered  activities. 
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- Define  usability  criteria  that  can  be  empirically  validated. 

5.2  Communication  among  stakeholders 

The  information  in  the  CISU-R  supports  communication  among  stakeholders  of  a product  to  obtain  a better 
understanding  of  the  usability  requirements  for  a product. 

For  example,  the  communication  can  be: 

- Among  members  of  a development  team,  to  specify  requirements  for  use  in  creating  or  updating  a 
product. 

- Between  the  customer  and  supplier  of  a custom  product,  to  define  specific  customer  requirements. 

- Between  potential  customers  and  a supplier  of  a commercially  or  publicly  available  product,  to  define 
diverse  requirements. 

Use  of  the  CISU-R  to  create  usability  requirements  helps  supplier  organizations  understand  what  the 
customer  wants  and  supports  the  pro-active  collaboration  between  a supplier  and  customer.  The  CISU-R 
provides  a mechanism  for  all  stakeholders,  in  either  supplier  or  customer  organizations,  to  consider  the 
requirements  before  design  begins,  and  can  reduce  the  need  for  later  redesign,  recoding,  and  retesting. 
Review  of  the  requirements  specified  in  the  CISU-R  can  reveal  misunderstandings  and  inconsistencies  early 
in  the  development  cycle  when  these  issues  are  easier  to  correct. 

Examples  of  how  information  in  the  CISU-R  can  be  used  include: 

- Customer  organizations  (or  customers  within  an  organization)  can  specify  usability  requirements  to 
accurately  describe  what  they  need. 

- Usability  professionals  and  designers  can  decide  whether  usability  requirements  specified  by  a 
customer  are  realistic,  and  to  plan  how  to  ensure  that  a product  meets  these  requirements. 

- Other  technical  professionals  and  managers  can  decide  whether  to  accept  usability  requirements. 

- Customer  organizations  can  include  usability  requirements  in  a contract  or  request  for  proposals  (or 
specify  that  they  be  created  and  approved  during  the  course  of  the  project.). 

- Stakeholders  in  customer  organizations  can  assess  whether  the  usability  requirements  meet  their 
needs. 

5.3  Benefits  of  usability  requirements 

Defining  usability  requirements,  and  using  them  as  a measure  of  the  success  of  a product  can  provide  many 
benefits  for  a product.  The  ultimate  benefit  is  in  increased  usability.  Benefits  for  the  product  and  development 
process  include: 

- Increasing  the  likelihood  of  product  success.  Specifying  performance  and  satisfaction  criteria 
derived  from  existing  or  competitor  systems  increases  the  likelihood  that  a product  will  meet  user 
needs,  and  greatly  reduces  the  risk  of  product  failure  as  a result  of  releasing  an  inferior  product. 

- Reducing  the  development  effort.  The  CISU-R  provides  a mechanism  for  the  various  concerned 
groups  in  the  customer’s  organization  to  consider  all  of  the  requirements  before  design  begins  and 
reduces  later  redesign,  recoding,  and  retesting.  Review  of  the  requirements  specified  in  the  CISU-R 
can  reveal  misunderstandings  and  inconsistencies  early  in  the  development  cycle  when  these  issues 
are  easier  to  correct. 

- Reducing  later  costs.  Identifying  usability  requirements  reduces  the  risk  of  unplanned  re-work  later 
in  the  development  process,  helping  to  ensure  that  the  cost  estimates  are  more  accurate. 


7 


NIST/iUSR  - CISU-R 


- Providing  a baseline  for  usability  criteria.  When  detailed,  quantitative  measures  are  included  in  the 
requirements,  they  provide  a baseline  against  which  the  success  of  a product  in  complying  with  the 
requirements  can  be  measured. 

- Serving  as  a basis  for  enhancement  and  iteratively  refining  the  product.  The  CISU-R  provides  a 
foundation  for  continued  production  evaluation  and  refinement. 

- Tracking  evolving  requirements  by  providing  a format  to  document  usability  requirements. 

5.4  Development  of  requirements 

The  CISU-R  facilitates  the  iterative  development  of  requirements.  The  three  parts  of  a requirement  can  be 
documented  with  increased  precision  as  more  information  is  gathered  about  the  key  user  groups  and  their 
needs. 

Within  each  level,  a product  team  might  develop  the  three  parts  of  a requirement  in  stages: 

- First  specifying  the  expected  context  of  use.  This  provides  information  on  users,  the  environment,  and 
the  intended  use  of  the  product  that  are  the  basis  for  usability  requirements.  This  information  can  be 
communicated  to  designers,  developers  or  customers  for  review  and  confirmation. 

- Next  performance  and  satisfaction  criteria  can  be  specified  for  defined  scenarios  of  use.  These  criteria 
are  used  as  input  to  the  design  process,  and  to  create  testable  measures  to  validate  whether  a 
product  meets  the  requirements. 

- Finally  the  procedure  for  a usability  test  for  the  requirements  can  be  specified. 

After  each  iterative  round  of  discussion  and  revision,  the  stakeholders  can  approve  the  usability  requirements 
for  the  product.  These  requirements  then  form  the  basis  for  continued  product  development  and  testing. 

5.5  Role  in  user  centered  design 

Specifying  and  testing  usability  requirements  are  part  of  the  wider  user  centered  design  process.  Two 
activities  that  are  part  of  a user  centered  design  process  are  necessary  to  ensure  that  appropriate  information 
is  included  in  the  usability  requirements: 

- Identify  stakeholders:  All  the  usability  requirements  need  to  be  based  on  knowledge  from  identified 
stakeholders,  including  all  users  groups  as  well  as  other  interested  parties. 

- Elicit  information  from  stakeholders:  Stakeholders  such  as  customers,  users,  suppliers,  and 
developers  can  be  encouraged  to  participate  in  the  requirements  process,  and  provide  information 
about  the  context  of  use  (users,  tasks,  and  environments),  and  quality  requirements. 

To  ensure  that  the  requirements  are  correct,  other  user  centered  design  activities  (such  as  competitive 
analysis,  interviews,  surveys,  focus  groups,  field  studies,  task  analysis,  benchmark  usability  testing  or  paper 
prototyping)  can  be  used  to  obtain  feedback  from  users  to  iteratively  refine  requirements. 

Usability  requirements  specified  with  the  CISU-R  are  an  important  part  of  a product  specification,  and  can  be 
used  with  more  detailed  requirements  for  features  of  the  user  interface  to  guarantee  usability,  (see  Annex  B). 
The  value  of  specifying  the  high  level  CISU-R  usability  requirements  is  that  they  relate  closely  to  business 
requirements  for  successful  use  of  a product  and  increased  productivity. 
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6.  Usability  requirements  specification 
6.1  General 

Usability  requirements  based  on  the  CISU-R  have  three  parts:  the  context  of  use,  the  performance  and 
satisfaction  criteria,  and  the  test  method.  All  usability  requirements  include  all  three  parts,  but  there  are  three 
levels  of  compliance.  Some  projects  may  choose  to  develop  requirements  iteratively,  reaching  higher  levels  of 
compliance  over  time.  As  more  information  is  gathered  about  requirements,  the  usability  requirements  for  the 
key  user  groups  can  be  documented  with  increased  precision  and  specificity  as  the  parties  involved  finalize 
and  approve  the  usability  requirements  for  the  product. 

The  context  of  use:  a description  of  the  intended  users,  their  goals,  associated  equipment  (including 
hardware,  software,  and  materials),  the  physical  and  social  environment  in  which  the  product  will  be  used,  and 
examples  of  scenarios  of  use.  (see  Clause  6. 1 ) 

- A complete  description  of  the  context  of  use  as  specified  in  Section  6.2  shall  be  included  in  all 
usability  requirements  (Levels  1, 2,  and  3).  Additional  detail  may  be  added  during  the  design  and 
development  process. 

Performance  and  satisfaction  criteria:  ways  in  which  the  usability  of  the  product  can  be  measured, 
specified  for  the  main  scenarios  of  use.  They  may  include  target  values,  or  may  specify  how  these  values  will 
be  identified  (see  Clause  6.3).  These  criteria  can  be  specified  to  three  levels  of  completeness  and  precision: 

- Level  1:  the  requirements  shall  identify  the  criteria  appropriate  for  successful  use  of  the  product,  and 
the  relative  importance  of  each  and  identifying  those  that  are  practical  to  measure  directly  (see  Clause 
6.3.1). 

- Level  2:  the  requirements  shall  specify  target  values  for  the  criteria,  or  a range  of  acceptable  values, 
(see  Clause  6.3.2). 

- Level  3:  the  requirements  shall  include  specific  values  for  measures  of  user  performance  and 
satisfaction  to  be  tested,  or  a method  of  determining  those  values  (see  Clause  6.3.3). 

The  test  method:  how  the  product  will  be  tested  to  determine  whether  the  usability  requirements  have  been 
met.  The  specification  of  the  test  method  may  include  only  an  identification  of  the  appropriate  test  methods, 
may  describe  a method  and  identify  the  test  context,  or  may  include  a complete  specification  of  a test  method 
(see  Clause  6.4).  These  criteria  can  be  specified  to  three  levels  or  detail: 

- Level  1 : the  requirements  shall  identify  the  types  of  methods  that  are  appropriate  for  the  product  and 
criteria  selected  (see  Clause  6.4.1). 

- Level  2:  the  requirements  shall  include  a preliminary  test  method  and  the  criteria  to  be  evaluated 
(see  Clause  6.4.2). 

- Level  3:  the  requirements  shall  include  a complete  test  protocol  for  a user  performance  test,  as 
specified  in  Annex  C (see  Clause  6.4.3). 

These  levels  are  intended  to  ensure  that  usability  requirements  can  be  developed  with  the  CISU-R  for  all 
types  of  projects,  from  the  smallest  or  most  informal  to  the  most  complex  or  formally  specified  products.  The 
three  levels  of  compliance  allow  usability  requirements  to  be  written  to  an  appropriate  level  of  completeness 
and  precision  for  the  product.  At  the  same  time,  they  encourage  all  projects  to  reach  at  least  Level  1 
compliance.  This  level  includes  all  three  parts  of  a requirement  (context  of  use,  criteria,  and  test  methods),  but 
allows  for  less  detailed  criteria  and  test  method  specifications.  Each  level  contributes  to  good  usability.  The 
CISU-R  allows  an  appropriate  level  of  compliance  to  be  selected  for  each  project. 

EXAMPLE  1 A project  to  update  the  shopping  cart  on  an  e-commerce  web  site  might  aim  for  Level  1 compliance  with 
a usability  requirement  to  increase  successful  checkouts  (task  completion).  The  planned  evaluation  methods  might  also 
include  actual  site  performance  as  measured  by  server  log  analysis,  a checklist  of  requirements,  or  usability  testing. 
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EXAMPLE  2 A project  to  develop  a customer  service  application  might  aim  for  Level  2 compliance  with  a usability 
requirement  that  at  least  90  % customer  service  representatives  who  have  completed  their  initial  training  can  complete  a 
new  customer  order  accurately,  to  be  tested  with  sample  scenarios  in  a lab  setting  or  under  simulated  field  conditions. 

EXAMPLE  3 A project  to  develop  a hand-help  field  service  measurement  tool  might  aim  for  Level  3 compliance  by 
including  a complete  protocol  for  a summative  usability  test,  and  further  refining  the  criteria  and  criterion  values.  . 

In  addition,  a usability  requirements  document  should  include  information  to  assist  in  maintaining  and  refining 
the  requirements.  This  information  includes: 

a)  Information  to  support  the  traceability  of  these  requirements,  including  their  source  and  process. 

b)  Document  version  control  information 


6.2  Context  of  use 

6.2.1.  General 

For  all  levels  of  compliance  with  this  standard,  the  context  of  use  shall  include  descriptions  of. 

a)  Stakeholders 

b)  User  groups 

c)  Goals  and  tasks 

d)  Technical  environment  (equipment) 

e)  Physical  and  social  environments 

f)  Scenarios  of  use  for  the  most  important  goals 

NOTE  Important  goals  can  include  infrequent  but  critical  or  exceptional  tasks 

6.2.2.  Stakeholders 

Usability  requirements  shall  include  a description  of  all  groups  of  people  who  are  identified  as  stakeholders, 
including: 

a)  A list  of  all  groups  who  have  a legitimate  interest  in  the  product  throughout  its  life  cycle. 

b)  Their  roles  and  interests  in  the  product. 

EXAMPLE  1.  Role:  marketing  product  manager;  interest:  business  needs  solutions 
EXAMPLE  2 Role:  user/administrative  assistant;  interest  facilitation  of  administrative  duties 

c)  If  any  of  the  stakeholders  will  not  be  considered  in  identifying  requirements,  the  rationale  for 
excluding  them. 

NOTE  Information  about  the  context  of  use  (users,  tasks,  and  environments)  and  requirements  for  usability  should  be 
elicited  from  the  stakeholders.  Some  stakeholders  needs  are  not  stated  explicitly,  but  implied  from  the  context  where  the 
product  is  to  be  used.  Often  stakeholders  needs  are  identified  from  analyzing  business  processes  or  how  existing  products 
are  used. 

6.2.3.  User  groups 

Usability  requirements  shall  include  a list  of  all  anticipated  user  groups. 
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Requirements  should  be  provided  for  the  user  groups  who  use  a product  most  frequently  or  who  are  critical  for 
business  purposes.  Key  characteristics  and  capabilities  of  these  primary  user  groups  shall  be  defined. 

a)  Characteristics  should  be  relevant  to  the  product  and  context  of  use. 

NOTE  Relevant  factors  may  include  experience  in  using  computers,  use  of  similar  products,  familiarity 
with  the  task,  frequency  of  usage,  expertise,  training,  culture,  nationality,  or  age. 

b)  Characteristics  should  be  defined  in  enough  detail  that  they  can  be  used  to  select  representative 
usability  test  participants. 

c)  Characteristics  should  include  identification  of  any  disabilities  that  may  affect  the  ability  to  use  the 
product. 

NOTE  Requirements  may  only  become  evident  when  a user  first  tries  to  apply  the  product  to  their  business 
process  or  task. 

6.2.4.  Goals 

The  main  goals  for  each  user  group  shall  be  listed,  without  reference  to  any  specific  means  of  achieving 
them.  The  goals  should  be  an  intended  outcome  of  value  to  the  user  or  business,  for  example:  accurately 
completing  a particular  form,  locating  the  most  relevant  information,  or  successfully  setting  up  a computer. 

NOTE  It  may  be  possible  to  identify  and  list  all  the  user  goals  for  some  products.  In  other  case,  the  list  may  be 
limited  to  the  most  important,  frequent  or  representative  goals. 

6.2.5.  Technical  environment  (equipment) 

Any  relevant  aspects  of  the  intended  or  anticipated  computing  or  other  technical  environment  shall  be 
specified. 

Examples  (particularly  applicable  to  software  products)  include: 

a)  Hardware  configuration:  e.g.,  processor  speed,  memory  size,  network,  storage,  input,  and  output 
devices. 

b)  Screen  type  (CRT  or  LCD),  resolution,  and  color  depth.  If  relevant,  also  include  display  (monitor)  size 
and  whether  multiple  monitors  are  required. 

c)  Print  media  size  and  print  resolution. 

d)  Whether  visual  interface  elements  (such  as  text)  can  vary  in  size  and  the  size(s)  available. 

e)  Software  configuration:  e.g.,  browser  version,  operating  system  version,  middleware,  database. 

f)  Assistive  technologies  available  to  users  with  disabilities. 

g)  Documentation  and  support  materials. 

EXAMPLE  Configuration  information  specific  to  a hardware  environment  might  include  the  wireless  signal  environment 
and  the  quality  of  connecting  cables. 

NOTE  These  are  only  examples.  For  custom  products  the  factors  may  be  quite  precise,  for  consumer  products  they 
may  specify  an  acceptable  range  or  anticipated  environment. 

6.2.6.  Physical  and  social  environments 

Any  aspects  of  the  expected  physical  and  social  environments  that  can  influence  usability  shall  be  specified: 


11 


NIST/IUSR  - CISU-R 


a)  Physical  environment  in  which  the  product  will  be  used,  including  location  and  relevant  physical 
conditions,  such  as  temperature  or  lighting. 

b)  Organizational  environment,  including  group  work  dynamics,  time  pressures,  supervision,  and 
support. 

c)  Any  physical,  health  or  safety  issues. 

d)  Any  possible  financial  or  security  risks 


6.2.7.  Scenarios  of  use 

The  goals  and  tasks  that  will  form  the  basis  of  the  requirements  shall  be  illustrated  with  scenarios  of  use. 
These  scenarios  describe  how  users  meet  their  goals.  They  are  examples  of  users'  activities,  motivations,  and 
how  they  carry  out  their  tasks,  using  the  product  in  a specific  situation. 

The  scenarios  should  illustrate  the  goals  to  be  met,  and  tasks  to  be  carried  out  to  meet  those  goals.  The 
scenarios  should  be  the  most  frequent  or  most  critical  to  the  business  or  user.  Each  scenario  includes  an 
identification  of  the  user  group,  technical,  physical,  and  social  environment,  the  goals  to  be  met  or  tasks  to  be 
carried  out,  and  the  relative  importance  of  the  scenario. 

There  may  be  separate  scenarios  for  first  use  (after  any  necessary  training)  and  experienced  use. 

6.2.8.  Training 

If  users  are  expected  to  study  any  documentation,  training  materials  or  courses  before  using  the  product,  the 
following  information  shall  be  specified: 

a)  Which  user  groups  are  expected  to  study  the  materials. 

b)  What  previous  knowledge  or  experience  is  required. 

c)  Which  goals  and  tasks  are  included  in  the  training  materials. 

6.3  Usability  performance  and  satisfaction  criteria 
6.3.1.  Identifying  relevant  usability  criteria  (Level  1) 

Performance  and  satisfaction  criteria  identify  the  measures  for  the  usability  of  a product.  The  first  level  in 
specifying  performance  and  satisfaction  criteria  is  to  identify  the  criteria  relevant  to  the  context  of  use  for  the 
product.  These  may  have  important  design  implications. 

The  following  information  shall  be  provided  for  Level  1 compliance: 

a)  The  types  of  performance  and  satisfaction  criteria  appropriate  for  successful  use  of  the  product. 

- These  might  include  performance  criteria  such  as  task  completion  rate  or  time  on  task,  subjective 
“ease  of  use”  scores,  indirect  measures  such  as  number  of  steps  in  a task,  or  a list  of  features  to 
support  usability  that  can  be  evaluated  with  a checklist. 

Different  user  groups  or  scenarios  may  have  different  criteria. 

b)  The  relative  importance  of  each  criteria  to  the  success  of  the  product 

Examples  of  performance  and  satisfaction  criteria  include:  the  unassisted  completion  rate  (effectiveness),  the 
mean  time  taken  to  successfully  complete  each  goal  (efficiency),  satisfaction  scores  on  a specific 
questionnaire,  efficiency  relative  to  a past  benchmark  or  other  product.  Annex  B provides  additional 
information  about  selecting  appropriate  criteria  and  best  practices  for  how  they  should  be  measured. 
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EXAMPLE  An  example  Level  1 specification  for  usability  criteria  from  Annex  F.6  is  “The  performance  criteria  for  Web 
Delivery  will  be  the  ability  of  appropriate  users  to  complete  all  core  tasks  with  no  training  or  assistance  except  on-screen 
prompts.  Satisfaction  with  Web  Delivery,  after  completing  these  tasks  must  be  at  least  as  high  as  satisfaction  scores  for 
current  delivery  methods." 

6.3.2.  Defining  target  values  for  criteria  (Level  2) 

The  second  step  in  specifying  performance  and  satisfaction  criteria  is  to  provide  target  values  for  the  criteria, 
or  indicate  a method  for  determining  those  values  later  in  the  project. 

Target  values  are  the  minimum  acceptable  value,  or  a range  of  acceptable  values,  for  the  criteria.  They  may 
be  based  on  the  value  for  an  existing  or  competitor  system  used  for  the  same  tasks  and  goals,  based  on 
expert  judgment  or  specified  by  stakeholders.  Additional  target  values  better  than  the  minimum  acceptable 
value  may  also  be  specified. 

For  Level  2 compliance,  the  usability  criteria  shall  include  target  values  or  a range  of  acceptable  values  for 
these  criteria.  Target  values  may  be  in  actual  numbers,  percentages,  average  or  means,  a range  of  values,  or 
a scale,  they  may  be  absolute,  or  relative  to  performance  benchmarks 

Each  target  value  may  be  given  as: 

a)  a definite  requirement,  or 

b)  a provisional  requirement  subject  to  further  negotiation,  or 

c)  an  objective  for  guidance. 

For  effectiveness,  efficiency,  and  satisfaction  criteria,  either  target  values  shall  be  given,  or  a note  provided 
explaining  that  either: 

a)  a target  value  will  be  provided  later,  or 

b)  why  a target  value  cannot  be  established  (for  example  for  a completely  new  product),  or 

c)  why  no  target  value  is  needed  (for  example  the  measure  is  of  low  importance  to  the  user  or 
organization). 

EXAMPLE  An  example  of  a Level  2 specification  for  usability  criteria  is  included  in  Annex  G.6.2.  It  updates  the  criteria 
to  add  target  values,  and  to  further  define  the  criteria  and  requirements  for  testing  them. 

6.3.3.  Defining  specific  criterion  values  (Level  3) 

For  Level  3 compliance  the  requirements  shall  include  specific  values  for  each  criterion.  These  may  be 
validated  by  benchmark  usability  testing,  business  requirements,  or  other  methods. 

NOTE  One  way  to  establish  criteria  values  is  to  first  measure  the  effectiveness,  efficiency,  and  satisfaction  for  the 
same  goals  using  an  existing  or  competitor  system,  and  then  to  set  the  requirements  for  the  new  system  to  be  as  good  as 
or  better  than  the  existing  system. 

EXAMPLE  The  example  Level  3 specification  for  usability  criteria  in  Annex  H.10  adds  detail  on  the  specific  values 
and  how  they  were  determined.. 

6.3.4.  Including  use  of  learning  materials  in  the  criteria 

If  any  documentation,  training  materials  or  course  will  be  studied  before  the  product  is  used,  criteria  for  the 
usability  of  the  training  materials  in  at  least  one  training  scenario  (effectiveness,  efficiency,  and  satisfaction 
when  completing  training  goals)  should  be  included. 

EXAMPLE  1 Given  2 hours  of  training  with  the  online  training  program,  a user  will  be  able  to  complete  the  item 
identification  task  under  1 minute  and  with  fewer  than  2 errors. 
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EXAMPLE  2 After  reading  the  owner’s  manual  a user  will  be  able  to  access  and  listen  to  a cellphone  message  on  the 
first  attempt. 

6.4  Methods  for  testing 

6.4.1.  Identifying  evaluation  methods  (Level  1) 

The  final  part  of  a usability  requirement  identifies  methods  to  evaluate  whether  the  criteria  have  been  met  and 
the  context  in  which  the  criteria  will  be  measured. 

The  first  level  in  specifying  test  methods  is  to  identify  methods  that  are  appropriate  for  the  product.  For  some 
products  these  methods  may  include  checklists  or  other  inspection  methods,  or  indirect  measures  such  as 
business  metrics,  but  ideally,  they  include  methods  in  which  people  who  are  representative  of  the  user  groups 
test  the  product. 

For  Level  1 compliance,  the  usability  requirements  shall  include  a list  of  usability  methods  that  can  be  used  to 
determine  whether  the  usability  requirements  have  been  met. 

NOTE  An  expert  review  against  a checklist  of  requirements  would  be  appropriate  for  requirements  that  specify 

the  presence  (or  absence)  of  a feature  or  elements  of  an  interactive  function,  such  as  an  undo  feature  or  conformance  to 
other  guidelines.. 

EXAMPLE  An  example  Level  1 specification  (from  Annex  F.7)  is  “The  product  will  be  evaluated  against  the 
performance  and  satisfaction  criteria  with  a usability  test,  using  test  scenarios  based  on  the  three  scenarios.” 

6.4.2.  Specifying  a preliminary  usability  evaluation  method  (Level  2) 

The  second  level  in  specifying  test  methods  is  to  describe  the  method  to  be  used  to  test  whether  the  usability 
requirements  have  been  met,  and  the  context  in  which  the  measurements  will  be  made.  This  level  adds 
enough  information  to  describe  the  basic  test  method  to  a usability  professional,  but  is  not  a complete 
specification  of  the  method. 

For  Level  2 compliance,  the  preliminary  evaluation  method  shall  include  a description  of  the  methods  used  to 
measure  the  product  on  all  of  the  criteria. 

If  a usability  test  method  is  specified,  the  description  shall  include: 

a)  Goals  of  the  test,  including  task  scenarios  to  be  tested  and  criteria  for  successful  completion  of  each 
goal  (see  Annex  C.2.1.a  and  b). 

b)  The  user  groups  (identified  in  the  context  of  use)  to  be  included  in  the  test  (see  Annex  C.1  .b). 

c)  The  type  of  test  facility,  including  the  setting  and  type  of  space,  and  how  relevant  aspects  of  the 
intended  environment  will  be  simulated  (see  Annex  C2.2). 

d)  Computing  environment,  which  may  be  specific  or  anticipated  (for  example,  specifying  the  “current 
version  at  the  time  of  the  test"  (see  C.2.3). 

e)  A general  description  of  the  test  procedure  (see  C.3. 1 ). 

EXAMPLE  An  example  Level  2 specification  (from  Annex  G.7)  adds  information  about  the  test  facility,  the  computing 
environment  and  display  devices,  as  well  as  a brief  outline  of  the  experimental  design. 

6.4.3.  Specifying  a complete  usability  test  method  (Level  3) 

For  level  3 compliance,  a full  protocol  for  a usability  test  shall  be  specified,  providing  the  information  required 
in  Annex  C. 
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The  specification  in  Annex  C is  consistent  with  ISO/IEC  25062  (Common  Industry  Format  (CIF)  for  usability 
test  reports),  which  should  be  used  for  reporting  the  results.  ISO/IEC  25062  requires  reporting  on  the  results 
of  measurements  of  efficiency,  effectiveness  and  satisfaction,  or  an  explanation  of  why  any  of  these  metrics 
are  not  meaningful  for  this  product. 

NOTE  The  performance  and  satisfaction  criteria  specified  in  6.3  establish  the  values  required  in  the  intended  context  of 
use.  The  accuracy  and  precision  with  which  they  are  measured  will  be  determined  by  the  context  of  measurement  (the 
number  of  and  types  of  users,  way  in  which  the  tasks  are  simulated,  and  the  simulated  working  environment). 

EXAMPLE  An  example  Level  3 specification  of  a complete  test  protocol  is  included  in  Annex  H.1 1 . 


6.5  Traceability 

The  following  information  should  be  documented  to  assist  the  development  organization  in  maintaining  and 
refining  the  requirements: 

a)  The  source  of  the  requirements. 

b)  How  the  usability  requirements  support  the  stakeholders’  business  objectives  for  the  product. 

c)  Why  the  goals  were  selected. 

EXAMPLE  The  most  frequent  goals,  the  most  troublesome  goals,  most  critical  to  accomplishing  business 
goals;  most  frequently  requested  requirement. 

d)  The  source  of  each  goal. 

EXAMPLE  Observation  of  customers  using  similar  products,  product  marketing  specifications 

e)  Who  requested  this  requirement. 

f)  Whether  the  requirement  is  essential  or  desirable. 

6.6  Document  control 

The  following  information  should  be  included  to  track  different  versions  of  the  requirements  document: 

a)  Version  number 

b)  Date  of  change 

c)  Nature  of  change 

d)  Person  making  the  change 

e)  Person  approving  the  change 
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Annex  A (informative) 
Usability  requirements  format 


A.1  Introduction 

This  annex  provides  a list  of  elements  that  should  be  used  in  a usability  requirements  document,  including 
document  and  product  information  elements.  It  may  either  be  used  as  a template,  or  may  be  used  as  a 
checklist  for  creating  a format  for  usability  requirements  created  with  the  CISU-R. 

The  format  of  the  information  in  this  annex  is  consistent  with  the  CIF  (ISO/IEC  25062). 


A. 2 Title  page 

The  title  page  contains: 

a)  Identification  of  the  requirements  as  conforming  to  Common  Industry  Specifications  for  Usability  - 
Requirements  v0.88. 

b)  The  name  of  the  product  and  version  that  the  requirements  are  for. 

c)  The  date  the  requirements  were  prepared. 

d)  The  organization  name. 

e)  Contact  details  for  questions  and/or  clarifications. 

f)  Names  of  the  people  who  produced  the  requirements. 

A. 3 Executive  summary 

The  Executive  Summary  provides  a high  level  overview  of  the  requirements.  The  intent  of  this  section  is  to 
provide  information  for  people  who  may  not  read  the  technical  body  of  this  document.  This  section  should 
begin  on  a new  page  and  end  with  a page  break  to  facilitate  its  use  as  a stand-alone  summary. 

The  executive  summary: 

a)  Explains  the  purpose  of  the  requirements. 

b)  States  the  level  of  conformance. 

c)  Provides  a high  level  overview  of  the  requirements  that  includes: 

Name,  version,  and  brief  description  of  the  product. 

Summary  of  the  type  of  users,  tasks,  any  associated  equipment  (hardware,  software,  and 
materials),  and  the  physical  and  social  environments  in  which  the  product  is  intended  to  be  used. 

- A list  of  the  task  scenarios,  with  criteria  for  effectiveness,  efficiency,  and  satisfaction  (for  level  2 
and  3 conformance). 

d)  Explains  the  reason  for  and  nature  of  the  requirements. 


16 


NIST/IUSR  - CISU-R 


A.4  Product  details 

A.4.1.  Product  description 

A product  description  includes: 

a)  The  formal  product  name  and  version. 

b)  The  parts  of  the  product  for  which  requirements  are  being  provided. 

c)  The  user  groups  for  which  the  product  is  intended. 

d)  Whether  the  product  is  intended  to  be  “walk  up  and  use",  or  whether  any  documentation,  training 
materials  or  course  has  to  be  studied  before  the  product  is  used. 

e)  The  type  of  user  work  that  is  supported  by  the  product. 

f)  Description  of  the  environment  in  which  the  product  should  be  used. 

g)  Support  for  any  groups  with  special  needs. 

A.4.2.  Product  objectives 

Product  objectives  include: 

a)  The  overall  objectives  for  the  product  and  specific  objectives  for  any  subset  of  usage. 

b)  Any  known  or  intended  functions  and  components  that  support  key  objectives. 

A.5  Context  of  use  and  scenarios 

The  context  of  use  and  scenarios  include  all  of  the  information  and  headings  specified  in  Clause  6.2: 

a)  Stakeholders 

b)  User  groups 

c)  Goals  and  tasks 

d)  Technical  environment  (equipment) 

e)  Physical  and  social  environments 

f)  Scenarios  of  use  for  the  most  important  goals 

A.6  Usability  performance  and  satisfaction  criteria  and  values 

The  usability  performance  and  satisfaction  criteria  include  the  information  specified  in  Clause  6.3,  including 

a)  The  goals  that  will  form  the  basis  of  the  requirements,  with  scenarios  of  usage,  with  an  explanation  of 
why  these  goals  and  scenarios  were  selected. 

b)  The  relative  importance  of  each  goal. 

c)  A definition  of  the  performance  and  satisfaction  criteria  appropriate  for  the  scenario  and  product. 
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d)  Target  values  or  a range  of  anticipated  values  for  these  criteria,  which  may  be: 
a definite  requirement,  or 

- a provisional  requirement  subject  to  further  negotiation,  or 

- an  objective  for  guidance. 

Criteria  may  include  target  values  for  effectiveness,  efficiency,  and  satisfaction,  an  explanation  that  a target 
value  will  be  provided  later,  or  an  explanation  of  why  a target  value  cannot  be  established  (for  example  for  a 
completely  new  product). 

If  any  documentation,  training  materials  or  course  has  to  be  studied  before  the  product  is  used,  measures 
should  be  given  for  the  usability  of  the  training  materials  in  at  least  one  training  scenario  (effectiveness, 
efficiency,  and  satisfaction  when  completing  training  goals). 


A. 7 Requirements  for  testing 

If  a protocol  for  a usability  test  is  included,  the  information  specified  in  Annex  C may  be  used. 
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Annex  B (Informative) 

Examples  of  performance  and  satisfaction  measures 


B.1  Introduction 

This  annex  provides  examples  of  performance  and  satisfaction  measures,  appropriate  for  use  in  usability 
requirements  created  with  the  CISU-R.  Because  the  CISU-R  uses  the  definition  of  usability  from  ISO  9241 
(see  the  definition  in  Clause  4.1),  these  examples  focus  on  effectiveness,  efficiency,  and  satisfaction 
measures. 

It  is  desirable  to  set  criteria  for  user  performance  and  satisfaction  criteria  that  can  be  measured  during 
development.  This  will  give  a high  degree  of  confidence  that  the  product  will  be  usable  in  the  tested  context  of 
use,  which  is  important  if  poor  user  performance  or  satisfaction  will  endanger  success  of  the  project  or 
product.  Sometimes  it  will  only  be  possible  to  measure  whether  the  criteria  have  been  achieved  after  the 
product  is  released  (for  example  from  observation  of  users,  or  from  web  server  logs).  This  will  provide 
feedback  on  the  adequacy  of  earlier  usability  work,  and  the  need  for  any  improvement  in  later  releases. 

If  any  documentation,  training  materials  or  course  will  be  studied  before  the  product  is  used,  criteria  for  the 
usability  of  the  training  materials  in  at  least  one  training  scenario  (effectiveness,  efficiency,  and  satisfaction 
when  completing  training  goals)  should  be  included. 

Other  types  of  usability  criteria  such  as  the  degree  of  adherence  to  usability  guidelines,  the  number  of  user 
actions,  or  an  expert  rating  can  also  provide  feedback  earlier  in  design  and  development.  However  they  do 
not  give  the  same  degree  of  confidence  in  final  usability. 


B.2  Effectiveness 

Effectiveness  relates  the  goals  of  using  the  product  to  the  accuracy  and  completeness  with  which  these  goals 
can  be  achieved.  Common  measures  of  effectiveness  include  task  completion  rate,  frequency  of  errors, 
frequency  of  assists  to  the  participant  from  the  testers.  It  does  not  take  account  of  how  the  goals  were 
achieved,  only  the  extent  to  which  they  were  achieved. 

B.2.1.  Task  completion  rate 

The  completion  rate  is  the  percentage  of  participants  who  completely  and  correctly  achieve  each  goal.  If  goals 
can  be  partially  achieved  (e.g.,  by  incomplete  or  sub-optimum  results)  then  it  may  also  be  useful  to  set 
requirements  for  partial  goal  achievement,  scored  on  a scale  of  0 to  100  % based  on  specified  criteria  related 
to  the  value  of  a partial  result. 

EXAMPLE  1 The  percentage  of  customers  who  can  successfully  complete  a transaction  on  a web  site,  or  the 
percentage  of  users  who  can  successfully  record  an  hour  long  TV  program  with  a Digital  Video  Recorder  (DVR). 

EXAMPLE  2 A spell-checking  task  might  involve  identifying  and  correcting  10  spelling  errors.  The  completion  rate 
might  be  calculated  based  on  the  percent  of  errors  corrected.  Another  method  for  calculating  completion  rate  is  weighting; 
e.g.,  spelling  errors  in  the  title  page  of  the  document  are  judged  to  be  twice  as  important  as  errors  in  the  main  body  of  text. 
The  rationale  for  choosing  a particular  method  of  partial  goal  analysis  should  be  stated,  if  such  results  are  included  in  the 
requirements. 

B.2.2.  Errors 

Although  the  requirements  for  effectiveness  are  based  on  error-free  completion  of  the  task,  keeping  track  of 
the  errors  is  particularly  helpful  in  determining  what  aspects  of  the  product  need  improvement.  Errors  are 
instances  where  test  participants  did  not  complete  the  task  successfully,  or  had  to  attempt  portions  of  the  task 
more  than  once.  Scoring  of  data  should  include  a classification  of  types  of  errors. 
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B.2.3.  Assists 

When  participants  cannot  proceed  on  a task,  the  test  administrator  sometimes  gives  direct  procedural  help  in 
order  to  allow  the  test  to  proceed.  This  type  of  intervention  is  called  an  assist  for  the  purposes  of  this  standard. 

If  it  is  necessary  to  provide  participants  with  assists,  efficiency,  and  effectiveness  metrics  shall  be  provided  for 
both  unassisted  and  assisted  conditions,  and  the  number  and  type  of  assists  shall  be  included  as  part  of  the 
test  results.  When  assists  are  allowed  or  provided,  the  number  and  type  of  assists  shall  be  included  as  part  of 
the  test  results. 

- The  unassisted  completion  rate  shall  be  calculated  as  the  percentage  of  participants  who  achieve  a 
goal  without  intervention  from  the  testers. 

- The  assisted  rate  shall  be  calculated  as  the  percentage  of  participants  who  achieve  a goal  only  with 
tester  intervention. 

EXAMPLE  For  example,  if  a participant  received  an  assist  on  Task  A,  but  went  on  to  successfully  complete  the  task 
following  the  assist,  he  could  be  included  in  the  assisted  Task  A completion  rate. 

In  some  usability  tests,  participants  are  instructed  to  use  support  tools  such  as  online  help  or  documentation, 
which  are  part  of  the  product,  when  they  cannot  complete  goals  on  their  own.  Accesses  to  product  features 
which  provide  information  and  help  are  not  considered  assists  for  the  purposes  of  this  requirements.  It  may, 
however,  be  desirable  to  measure  the  frequency  of  accesses  to  different  product  support  features,  especially 
if  they  affect  participants’  ability  to  use  products  independently. 


B.3  Efficiency 
B.3.1.  Measures 

Efficiency  relates  the  level  of  effectiveness  achieved  to  the  quantity  of  resources  expended.  Efficiency  is 
generally  assessed  by  the  mean  time  taken  to  achieve  the  task.  Efficiency  may  also  relate  to  other  resources 
(e.g.  total  cost  of  usage).  A common  measure  of  efficiency  is  time  on  task,  which  can  be  defined  as  the  mean 
time  taken  to  complete  each  task,  together  with  the  range  and  standard  deviation  of  times  across  participants. 

B.3.2.  Relative  user  efficiency 

Relative  user  efficiency  is  the  mean  time  taken  by  users  who  successfully  achieve  a goal  divided  by  the  time 
taken  by  an  expert. 

NOTE  Task  time  values  are  useful  when  making  comparisons  between  products,  while  the  relative  user  efficiency 
values  are  easier  to  interpret  in  isolation  because  they  compare  performance  in  the  usability  test  with  what  is  possible  by 
an  expert. 

B.3. 3.  Completion  rate/mean  time-on-task 

Completion  Rate  divided  by  Mean  Time-On-Task  is  the  core  measure  of  efficiency.  It  is  the  percentage  of 
users  who  were  successful  (or  percentage  goal  achievement)  for  every  unit  of  time. 

This  formula  shows  that  as  the  time  on  task  decreases,  one  would  expect  users  to  be  more  successful.  A very 
efficient  product  has  a high  percentage  of  successful  users  in  a small  amount  of  time.  This  allows  customers 
to  compare  fast  error-prone  interfaces  to  slow  easy  interfaces. 

EXAMPLE  1 An  error-prone  interface  such  as  a command  line  interface  using  wildcards  to  delete  groups  of  files  would 
typically  show  a low  completion  rate  divided  by  mean  time-on-task. 

EXAMPLE  2 A slower,  but  easier  interface  such  as  using  a mouse  and  keyboard  to  drag  each  file  to  the  trash  would 
typically  show  a high  completion  rate  divided  by  mean  time-on-task. 
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B.4  Satisfaction 

Satisfaction  describes  a user’s  subjective  response  when  using  the  product.  User  satisfaction  may  be  related 
to  whether  a users  wants  use  a product  and  may  affect  performance  in  some  cases.  Questionnaires  to 
measure  satisfaction  and  associated  attitudes  often  use  Likert  or  semantic  differential  scales. 

A variety  of  instruments  are  available  for  measuring  user  satisfaction  of  interactive  products,  and  many 
organizations  create  their  own.  Some  examples  of  standard  questionnaires  include:  ASQ  [5],  CUSI  [6], 
PSSUQ  [6],  QUIS  [3],  SUMI  [4],  and  SUS  [7]). 
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Annex  C (normative) 
Specification  of  a usability  test  protocol 


C.1  Introduction 

This  annex  provides  the  information  necessary  to  specify  the  method  and  test  protocol  to  test  whether  the 
usability  requirements  have  been  met.  The  elements  specified  are  consistent  with  the  ISO/IEC  25062,  the 
Common  Industry  Format  for  Usability  Test  Reports  (CIF),  and  the  results  of  the  test  can  subsequently  be 
documented  using  the  CIF  (ISO/IEC  25062). 

For  specifying  a usability  test  method  to  Level  3 compliance,  all  of  the  information  specified  in  this  annex  shall 
be  included. 

Sufficient  information  shall  be  provided  to  enable  the  requirements  to  be  tested. 


C.2  Users 

The  following  information  shall  be  provided: 

a)  The  total  number  of  participants  required. 

b)  Segmentation  of  user  groups  tested,  if  more  than  one. 

c)  How  participants  are  to  be  selected  so  that  they  have  the  essential  characteristics. 

Characteristics  should  be  chosen  to  be  relevant  to  the  product’s  usability.  They  should  be  sufficiently  detailed 
to  ensure  that  the  participants  are  representative  of  the  actual  users. 

The  following  information  should  be  provided: 

a)  Any  expected  differences  between  the  participant  sample  and  the  user  population. 

EXAMPLE  Actual  users  will  attend  a training  course  which  will  have  to  be  simulated. 

b)  Description  of  any  groups  with  special  needs. 

C.3  Context  of  product  use  in  the  test 
C.3.1.  Test  facility 

The  following  information  should  be  provided: 

a)  The  setting,  and  type  of  space  in  which  the  evaluation  will  be  conducted. 

EXAMPLES  Usability  lab,  cubicle  office,  meeting  room,  home  office,  kitchen,  manufacturing  floor 

b)  How  relevant  aspects  of  the  intended  environment  will  be  simulated. 

c)  How  the  test  setting  differs  from  the  normal  context  of  use. 


22 


NIST/IUSR  - CISU-R 


C.3.2.  Participant’s  computing  environment 

If  the  product  runs  on  a computer  or  other  device,  the  technical  environment  to  be  used  shall  be  specified, 
including: 

a)  Computer  configuration,  including  model,  operating  system  version,  required  libraries  or  settings. 

b)  If  used,  browser  name  and  version;  relevant  plug-in  names  and  versions. 

c)  Any  relevant  physical  connectors  or  wireless  connections. 

C.3.3.  Display  devices 

If  a display  device  is  part  of  the  product,  the  characteristics  of  the  display  shall  be  specified: 

a)  If  screen-based,  screen  size,  resolution,  and  color  setting. 

b)  If  print-based,  the  media  size  and  print  resolution. 

c)  If  visual  interface  elements,  such  as  fonts,  can  vary  in  size,  specify  the  size(s)  to  be  used. 

C.4  Test  procedure 

C.4.1.  Scenarios  to  be  tested 

The  following  information  shall  be  provided: 

a)  If  more  than  one  condition  is  being  tested,  the  logical  design  of  the  test. 

b)  Scenarios  to  be  tested. 

c)  Criteria  for  successful  completion  of  the  goal  of  each  scenario. 

d)  Maximum  time  limits  for  completing  a scenario  or  task. 

e)  Policies  and  procedures  for  interaction  between  tester(s)  and  test  participants. 

C.4. 2.  Participant  general  instructions 

The  following  information  shall  be  provided  (here,  or  in  an  annex): 

a)  Orientation  to  the  test  context  and  consents  to  be  given  to  the  participants. 

b)  General  instructions  to  be  given  to  the  participants. 

c)  Instructions  on  how  participants  are  to  interact  with  any  other  persons  present,  including  how  they  are 
to  ask  for  assistance  or  interact  with  other  participants,  if  applicable. 

C.4.3.  Participant  task  instructions 

The  following  information  shall  be  provided: 

a)  Task  instructions  to  be  given  to  the  participants. 

b)  Any  data  to  be  given  to  the  participants  to  enable  them  to  complete  the  scenarios. 
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C.5  Performance  and  satisfaction  metrics 

C.5.1.  Criteria  and  measurements 

The  following  information  shall  be  provided: 

a)  The  performance  criteria  to  be  measured. 

b)  How  performance  criteria  will  be  measured. 

c)  Definitions  of  any  independent  variables  or  control  variables. 


C.5.2.  Metrics  for  effectiveness,  efficiency,  and  satisfaction 

The  metrics  to  be  measured  and  reported  shall  include  either: 

a)  At  least  one  of  the  criteria  for  effectiveness,  and  efficiency,  and 

b)  The  questionnaire  or  other  method  for  measuring  satisfaction,  or 

c)  An  explanation  of  why  any  of  these  metrics  are  not  meaningful  for  this  product. 

EXAMPLE  Suppose  that  the  context  of  use  for  the  product  includes  real  time,  open-ended  interaction  between 
close  associates.  In  this  case,  Time-On-Task  may  not  be  meaningfully  interpreted  as  a measure  of  efficiency, 
because  for  many  users,  time  spent  on  this  task  is  "time  well  spent". 

C.5. 3.  Statistical  significance 

The  CIF  (ISO/IEC  25062)  specifies  the  statistical  data  to  be  given  when  reporting  results. 

A requirement  may  be  included  to  demonstrate  a relationship  with  previous  test  results,  for  example: 

- The  test  results  for  specified  measures  are  at  least  as  good  as  for  the  previous  results  (with  95  % 
confidence). 

- The  test  results  for  specified  measures  are  better  than  the  previous  results  (with  80  % confidence). 

C.6  Appendices 

The  following  information  may  be  provided: 

a)  Custom  questionnaires,  if  used. 

b)  Participant  general  instructions  (if  not  in  the  body  of  the  requirements). 

c)  Participant  task  Instructions. 
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Annex  D (informative) 

An  example  of  a user  centered  design  process 


A typical  user-centered  development  process  that  incorporates  the  development  of  usability  requirements 
under  the  CISU-R  might  include  the  following  subsequent  activities: 

a)  Define  scenarios  of  use. 

b)  Document  the  context  of  use  and  scenarios  of  use  in  CISU-R  format  (level  1 compliance). 

c)  Carry  out  a usability  test  of  an  existing  system  to  obtain  baseline  measures  of  effectiveness,  efficiency 
and  satisfaction  in  these  scenarios. 

d)  Use  the  baseline  data  as  a basis  for  establishing  quantitative  requirements  for  effectiveness, 
efficiency  and  satisfaction  in  these  scenarios  (level  2 compliance). 

e)  Specify  methods  to  test  whether  the  requirements  have  been  met  (level  3 compliance). 

f)  Refine  the  design  by  testing  with  users  carrying  out  tasks  using  low-fidelity  prototypes. 

g)  Test  prototypes  of  the  working  system. 

h)  Test  the  usability  of  the  final  system  to  check  whether  the  requirements  for  effectiveness,  efficiency, 
and  satisfaction  have  been  met.  The  results  can  be  documented  using  the  CIF  (ISO/IEC  25062) 
format. 

Depending  on  the  development  environment,  requirements  may  be  iteratively  elaborated  as  more  information 
is  obtained  from  usability  activities  such  as  paper  prototyping  during  development,  or  they  may  be  agreed  by 
all  parties  before  development  commences,  and  subsequently  only  modified  by  mutual  agreement. 

NOTE  The  requirements  for  effectiveness,  efficiency,  and  satisfaction  in  particular  contexts  of  use  are  less  likely  to 
change,  as  they  are  closely  linked  to  meeting  business  and  user  needs. 
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Annex  E (Informative) 

Examples  of  other  types  of  user  and  usability  requirements 

E.1  Introduction 

Other,  more  detailed  usability  requirements  can  be  specified  to  complement  the  core  CISU-R  requirements  in 
a user  centered  design  process  Examples  include: 

E.1.1.  Design  guidance 

a)  Design  principles 

b)  Human  factors  and  ergonomics 

c)  Style  guides 

d)  Standards 

E.1. 2.  Usability  features 

a)  Accessibility 

b)  Understandability 

c)  Learnability 

d)  Operability 

e)  Attractiveness 

E.1. 3.  Content  and  functions 

a)  Functionality 

b)  Content 

c)  Complete,  accurate,  up  to  date,  style 

d)  Effectiveness  (for  learning) 

e)  Trust 

f)  Platform  independence 


26 


NIST/IUSR  - CISU-R 


Annex  F (Informative) 

An  example  of  a Level  1 requirements  document 


F.1  Title  Page 

Usability  Requirements 

for  Web  Software  Delivery  Release  1 

ClFWorks 


15  May  2006 

Requirements  produced  by:  Sherman  Jones. 

Contributors:  Fred  Smith,  Product  Manager 

Reviewers:  Ann  Smith,  VP  of  Ul 

Alexander  Kushenko,  Product  Boss 

Created  under  the  Common  Industry  Specification  for  Usability  - Requirements  (CISU-R)  v0.88 

Address  inquiries  to:  Sherman  Jones,  ClFWorks 

Phone:  650-555-1212 

Email:  sherman.jones@CIFWorks.com 

Address:  150  New  Parkway,  New  Pines,  CA  9101 1 

Version  History 

3 May  2006  1.0  Sherman  Jones  Created  document 

15  May  2006  1.1  Sherman  Jones  Revised  based  on  review  comments 

Related  Documents 

"New  Product  Initiation  Plan”  - 13  April  2006  - Alexander  Kushenko 


F.2  Executive  Summary 

The  purpose  of  this  document  is  to  specify  the  usability  requirements  for  the  first  release  of  Web  Software 
Delivery  in  a format  that  allows  ClFWorks’  customers  to  review  and  provide  input.  This  document  conforms  to 
Level  1 of  the  CISU-R. 

The  objective  of  the  Web  Software  Delivery  product  is  to  move  away  from  physical  shipments  of  CD's  and 
DVD’s  towards  web  based  delivery  of  ClFWorks’  products.  The  key  tasks  involved  are  downloading  the 
software,  creating  DVDs  from  the  software  and  setting  up  a staged  (does  not  use  DVDs)  installation  from  the 
downloaded  files. 

The  intended  users  of  Web  Software  Delivery  are  system  administrators.  System  administrators  are 
experienced  computer  professionals  whose  primary  roles  are  the  installation,  configuration,  and  administration 
of  multi-user,  production  software  packages.  They  will  use  the  product  in  a corporate  computing  environment 
on  a computer  with  a recent  version  of  the  browser,  sufficient  disk  space  to  download  the  product  and  a high 
speed  (lOmbs  or  faster)  internet  connection. 

The  key  tasks,  for  which  requirements  scenarios  will  be  developed  are: 


Download  SITE  guard  software 
Burn  SITE  guard  software  to  DVD 
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- Prepare  SITE  guard  software  for  staged  install  and  launch  installer 

The  usability  of  this  product  will  be  evaluated  in  a usability  test  before  release,  measuring  the  following 
criteria.  Target  values  will  be  set  through  benchmark  usability  testing  of  the  current  delivery  methods, 
interviews  with  users  to  establish  their  requirements,  and  competitive  research  with  competitive  products. 


- Effectiveness:  the  unassisted  task  completion  rate,  measured  as  a percentage  of  total  task  attempts 
(priority  1) 

- Efficiency:  the  average  time  to  complete  the  task,  measured  as  the  average  number  of  minutes  to 
complete  each  task  (priority  2) 

- Satisfaction,  using  a standard  questionnaire.  This  may  be  the  SUMI,  SUS  or  a customized 
questionnaire  (priority  3) 


F.3  Product  Details 


Name  and  Version: 

Parts  for  which  requirements  provided: 
Intended  user  groups: 

Training  required: 

Work  supported  by  product: 

Usage  environment: 

Users  with  special  needs: 


ClFWorks  Web  Software  Delivery  Release  1 
All  parts  of  the  software 

System  Administrator,  experienced  with  server  software  installation 
None  (for  intended  user  groups) 

Downloading  and  preparing  ClFWorks’  Site  Guard  software  for 
installation. 

Medium  to  large  size  corporate  and  public  sector  companies. 

Product  will  meet  standards  for  Section  508  of  the  Rehabilitation  Act 


F.4  Product  Objective 

The  objective  of  the  product  is  to  replace  mailing  DVD’s  as  the  default  mechanism  for  shipping  ClFWorks' 
software. 


F.5  Context  of  Use 

F.5.1.  Users 

F. 5.1.1.  System  Administrators 

System  Administrators  are  experienced  computer  professionals  whose  primary  roles  are  the  installation, 
configuration,  and  administration  of  multi-user,  production,  software  . These  users  are  typically  administrators 
of  other  enterprise  software  products  such  as  operating  systems,  databases,  and  web  servers. 

System  Administrator’s  educational  backgrounds  are  varied  as  some  receive  formal  academic  training  in 
computer  science  or  computer  systems  management  while  others  receive  training  on  the  job  or  through 
vendor  or  3rd  party  courses. 

The  most  important  characteristic  of  a System  Administrator  is  on  the  job  experience.  There  are  typically  two 
routes  by  which  these  users  gain  experience  as  a System  Administrator: 
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g)  They  extend  their  background  from  being  a PC  administrator  or  power  user  into  the  role  of  a System 
Administrator  by  industry  training, 

h)  They  grow  their  skills  in  an  organization  under  the  tutelage  of  a Senior  System  Administrator. 

For  Web  Software  Delivery  we  expect  System  Administrators  to  have  1-10  years  of  experience  in  their  job.  In 
addition  to  software  installation,  and  configuration,  their  day  to  day  tasks  would  include  operations  (adding 
users,  adding  disk  space,  monitoring  an  existing  system),  and  troubleshooting.  They  may  also  perform 
planning  operations  for  future  software  and  hardware  acquisitions. 


F.5.1.2.  Users  with  Disabilities: 

Users  with  vision,  hearing,  and  motor  disabilities  should  be  able  to  function  as  a System  Administrator  with 
appropriate  assistive  devices  (e.g.  screen  readers). 

F.5.2.  Goals 

The  general  goal  is  to  obtain  the  appropriate  version  of  the  ClFWorks  Site  Guard  software  for  the  appropriate 
platform  in  a form  ready  for  installation.  Obtaining  the  software  completes  the  purchasing  process  but  may  or 
may  not  lead  immediately  to  installation.  In  some  cases  hardware  or  software  has  to  be  configured  before 
installation  of  the  software  begins.  Users  may  immediately  burn  the  electronic  version  of  the  software  onto 
DVD's,  however  this  is  not  required  for  installation.  Some  users  will  only  install  the  product  from  the 
downloaded  files  (referred  to  as  a “staged  install”).  Based  on  this  there  are  three  tasks  associated  with 
electronic  delivery. 

Task  1.  Finding  and  downloading  the  Site  Guard  software  via  Web  Software  Delivery 
Task  2.  Create  DVD’s  from  the  downloaded  Site  Guard  software  and  begin  the  install. 

Task  3.  Prepare  the  software  for  an  electronic  or  “staged”  install  and  begin  the  install 


F.5.3.  Equipment 

To  use  Web  Software  Delivery  the  following  computing  environment  will  be  required. 

- A computer  capable  of  running  the  browsers  listed  below. 

- Internet  Explorer  (6.0  and  higher),  Netscape  (6.0  and  higher),  or  Mozilla  Firefox  1.1  and  higher 

- 10  gigabytes  of  disk  space  to  download  the  zip  files  required  for  installation  and  create  the  staged 
installation. 

- 500  megabytes  of  RAM 

- A lOmbs  or  faster  internet  connection. 

- A SVGA  compatible  monitor. 

- A DVD  Burner 

F.5.4.  Physical  and  Social  Environment 

- The  physical  environment  will  be  either  a corporate/  public  sector  office  or  a home  office  with  a high 
speed  internet  connection  (10  mbs  or  faster).  The  home  office  may  be  used  to  download  the  software 
and  burn  CD’s  after  hours. 

- Typically  system  administrators  multi-task.  It  is  expected  that  using  web  software  delivery  will  be 
mixed  in  with  other  support  tasks  they  are  completing. 
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- There  is  typically  a time  lag  between  acquiring  and  installing  the  software,  and  rolling  out  the  software 
into  production.  Because  of  that  acquiring  and  installing  the  software  will  likely  be  part  of  a planning 
exercise  without  mission  criticality  pressure.  However  due  to  hacker  attacks  or  other  security 
concerns  there  may  be  times  when  obtaining  Site  Guard  may  be  urgent. 

F.5.5.  Scenarios  of  use 

Scenario  1:  Download  the  Site  Guard  software  and  burn  it  to  DVD 

Ann  is  the  lead  system  administrator  at  Biagra  a producer  of  genetically  engineered  wheat  products.  Last 
month  her  company  was  hit  by  a denial  of  service  attack  by  a group  of  organic  fundamentalists.  It  was  a 
rough  month  and  Ann  ended  up  shutting  down  the  company  website  leading  to  unhappy  customers  and 
employees.  It  took  no  time  to  get  the  PO  approved  for  Site  Guard  and  now  she  is  on  the  hot-seat  to  get  it 
installed  on  all  the  companies  servers.  She  is  glad  she  can  get  the  software  online  since  she  does  not 
want  to  wait  for  a physical  shipment.  Since  her  company’s  firewall  is  closed,  she  has  to  download  the 
software  at  home  and  take  it  into  the  office  and  install  it  on  her  2 primary  servers. 

After  dinner  she  turns  on  Toy  Story  2 for  the  kids  and  sits  down  in  the  study  to  download  the  software. 
She  has  the  printed  out  email  from  ClFWorks  and  uses  it  to  login  to  the  Web  Delivery  site.  After  login, 
she  sees  the  software  she  purchased  and  online  instructions  for  how  to  download  it.  She  uses  Mozilla  for 
the  download  as  it  has  a built  in  download  manager.  She  begins  the  first  DVD  download  and  then  goes 
to  check  on  the  kids.  Occasionally  she  peeks  in  on  the  computer  to  see  how  the  download  is  going.  She 
is  happy  to  see  that  it  is  progressing  nicely,  although  it  is  still  a bit  slow.  10  minutes  later  the  first  DVD  is 
downloaded  so  she  starts  the  second  DVD  download.  20  minutes  later  she  has  put  the  kids  to  bed  and 
comes  back  in  to  see  that  the  second  DVD  download  is  complete. 

Going  back  to  the  ClFWorks  website  she  launches  a utility  that  will  help  her  burn  the  DVD's.  It  asks  her 
for  her  computer  information  and  then  asks  her  to  insert  a DVD.  While  the  DVD  is  burning  she  sees  a 
progress  indicator.  It  says  20  minutes  to  go,  so  she  surfs  the  internet  for  a while.  Twenty  minutes  later 
she  starts  the  second  DVD.  At  the  end  of  the  evening  she  has  two  DVD’s  ready  for  installation  in  the 
morning. 


Back  at  work,  she  takes  the  DVD’s  and  pops  them  into  her  development  environment.  The  development 
environment  is  a clone  of  the  production  web  server  that  she  uses  to  test  out  installations.  She  opens  a 
Unix  Shell  and  launches  the  installer.  She  is  glad  to  see  that  the  installation  starts  successfully. 

Scenario  2:  Download  the  Site  Guard  software  and  create  a staged  installation 

Winston  is  a Senior  System  Administrator  for  the  University  of  South-East  North  Dakota.  He  administers 
the  systems  that  run  the  school’s  websites.  He  takes  pride  in  maintaining  a safe  system  despite  the 
many  hackers  at  the  school.  As  part  of  his  2005  budget,  he  got  approval  to  purchase  Site  Guard  and 
deploy  it  to  the  server  farm  that  is  used  for  the  university  website. 

Today  he  received  the  email  from  ClFWorks  with  his  license  number  and  instructions  for  downloading  the 
software.  Winston  always  likes  to  install  software  from  a staged  environment.  This  makes  it  much  faster 
to  take  a master  copy  of  the  software  and  install  it  on  multiple  machines  in  his  server  farm.  He  has 
designated  a system  called  “stage_guard"  to  be  the  stage  for  Site  Guard.  He  prints  out  the  email  (he 
can’t  access  it  from  stage_guard)  and  walks  over  to  the  stage_guard  machine.  He  logs  onto  the 
ClFWorks  website.  He  sees  the  software  that  he  needs  to  download  and  begins  the  first  download. 
While  the  first  download  is  completing  he  looks  around  on  the  website  for  instruction  on  building  a staged 
install.  He  sees  a link  to  instructions  for  staged  install  on  the  side  of  the  web  page.  He  clicks  the  link  and 
starts  reading  the  instructions.  The  instructions  say  he  can  either  do  it  manually  or  download  a script. 
Being  security  conscious  he  does  not  trust  "foreign”  scripts  so  he  does  it  the  manual  way. 

He  follows  the  instructions  and  begins  creating  the  directories  he  will  need  for  the  install.  It  takes  a while 
to  create  all  the  directories,  as  he  has  to  create  them  using  command  line  UNIX  and  then  set  privileges 
for  them.  He  checks  back  on  the  first  download  and  sees  it  is  complete.  He  starts  the  second  download. 
While  that  is  working  he  completes  the  directories  he  needs  and  focuses  on  unzipping  the  files.  The 
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instructions  say  to  use  a zip  utility  to  unpackage  and  put  the  files  in  the  proper  directories.  He 
appreciates  the  fact  that  the  instructions  list  the  exact  command  to  use  with  his  zip  utility.  He  unzips  the 
files  from  the  first  download  and  puts  them  into  the  appropriate  directory.  After  that  is  complete  he 
checks  back  in  with  the  download  and  sees  the  second  download  is  complete.  He  unzips  that  file  and 
puts  it  into  the  second  directory.  He  goes  back  to  the  first  directory  and  runs  the  installer.  He  is  glad  to 
see  it  has  come  up  and  he  can  proceed  with  the  installation. 


F.6  Performance  and  Satisfaction  Criteria 
F.6.1.  Goals 

The  goal  for  using  Web  Software  Delivery  is  to  find  and  download  the  software  for  the  appropriate  platform  in 
a form  ready  for  installation.  This  goal  is  completed  when  the  actual  installation  begins,  or  is  ready  to  begin. 

There  are  three  core  tasks  to  complete  this  goal  with  Web  Software  Delivery. 

1.  Download  SITE  guard  software 

This  task  has  the  highest  priority  since  successful  download  of  the  software  is  a prerequisite 
to  the  other  tasks. 

2.  Burn  SITE  guard  software  to  DVD  and  launch  installer 

This  task  has  second  priority  since  it  is  the  recommended  approach  for  installing  the  software. 

3.  Prepare  SITE  guard  software  for  staged  install  and  launch  installer 
This  task  is  lowest  priority  as  it  is  an  optional  way  of  installation. 

F.6.2.  Criteria 

The  performance  criteria  for  Web  Delivery  will  be  the  ability  of  appropriate  users  to  complete  all  core  tasks 
with  no  training  or  assistance  except  on-screen  prompts.  Satisfaction  with  Web  Delivery,  after  completing 
these  tasks  must  be  at  least  as  high  as  satisfaction  scores  for  current  delivery  methods. 

Target  values  for  these  criteria  will  be  set  through  interviews  with  users  to  establish  their  requirements,  results 
of  formative  usability  testing  during  the  design  process,  and  competitive  research  with  competitive  products. 


F.7  Methods  for  Testing 

The  product  will  be  evaluated  against  the  performance  and  satisfaction  criteria  with  a usability  test,  using  test 
scenarios  based  on  the  three  scenarios. 
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Annex  G (Informative) 

An  example  of  a Level  2 requirements  document 


G.1  Title  Page 

Usability  Requirements 

for  Web  Software  Delivery  Release  1 

ClFWorks 


25  August  2006 

Requirements  produced  by:  Sherman  Jones. 

Contributors:  Fred  Smith,  Product  Manager 

Reviewers:  Ann  Smith,  VP  of  Ul 

Alexander  Kushenko,  Product  Boss 
James  Dearing,  Director  of  Marketing 
Jennifer  Taylor,  Deployment  Manager 
John  Smith,  User  Researcher 
Jackie  Mayer,  Usability  Lab  Manager 

Created  under  the  Common  Industry  Specification  for  Usability  - Requirements  (CISU-R)  v0.88 


Address  inquiries  to: 
Phone: 

Email: 

Address: 


Sherman  Jones,  ClFWorks 
650-555-1212 

sherman.jones@CIFWorks.com 

150  New  Parkway,  New  Pines,  CA  91011 


Version  History 

10  May  2006 

1.0 

Sherman  Jones 

15  May  2006 

1.1 

Sherman  Jones 

10  July  2006 

2.0 

Sherman  Jones 

18  August  2006 

2.1 

John  Smith 

25  August  2006 

2.2 

Sherman  Jones 

Created  document 

Revised  based  on  review  (Level  1 complete) 
Added  detail  for  criteria 
Added  preliminary  test  method 
Added  target  values  (Level  2 complete) 


Related  Documents 

’’New  Product  Initiation  Plan”  - 1 3 April  2006  - Alexander  Kushenko 
’’Product  competitive  landscape  review”  - 8 July  2006  - James  Dearing 
’’Site  visits  and  user  observations  report"  - 5 August  2006  - John  Smith 
"Product  specifications  document”  - 10  August  2006  - Fred  Smith 
’’User  interface  specifications”  - 15  August  2006  - Ann  Smith 


G.2  Executive  Summary 

The  purpose  of  this  document  is  to  specify  the  usability  requirements  for  the  first  release  of  Web  Software 
Delivery  in  a format  that  allows  ClFWorks’  customers  to  review  and  provide  input. 

This  document  conforms  to  Level  2 of  the  CISU-R.  It  adds  to  the  previous  Level  1 document  with: 
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- Complete  definitions  for  performance  and  satisfaction  criteria 

- Preliminary  target  values,  to  be  confirmed  with  benchmark  usability  testing 

- Preliminary  usability  testing  method  to  test  the  product  against  the  criteria 

The  objective  of  the  Web  Software  Delivery  product  is  to  move  away  from  physical  shipments  of  CD’s  and 
DVD's  towards  web  based  delivery  of  ClFWorks’  products.  The  key  tasks  involved  are  downloading  the 
software,  creating  DVDs  from  the  software  and  setting  up  a staged  (does  not  use  DVDs)  installation  from  the 
downloaded  files. 

The  intended  users  of  Web  Software  Delivery  are  system  administrators.  System  administrators  are 
experienced  computer  professionals  whose  primary  roles  are  the  installation,  configuration,  and  administration 
of  multi-user,  production  software  packages.  They  will  use  the  product  in  a corporate  computing  environment 
on  a computer  with  a recent  version  of  the  browser,  sufficient  disk  space  to  download  the  product  and  a high 
speed  (lOmbs  or  faster)  internet  connection. 

The  key  tasks,  for  which  requirements  scenarios  have  been  developed  are: 

- Download  SITE  guard  software 

- Burn  SITE  guard  software  to  DVD 

- Prepare  SITE  guard  software  for  staged  install  and  launch  installer 

The  usability  of  this  product  will  be  evaluated  in  a usability  test  before  release,  measuring  the  following 
criteria.  This  document  includes  preliminary  target  values,  which  will  be  confirmed  with  a benchmark  usability 
test  before  final  testing  of  the  product. 

- Effectiveness:  the  unassisted  task  completion  rate,  measured  as  a percentage  of  total  task  attempts 


(priority  1 ) 


- Efficiency:  the  average  time  to  complete  the  task,  measured  as  the  average  number  of  minutes  to 
complete  each  task  (priority  2) 

- Satisfaction,  using  a standard  questionnaire.  This  may  be  the  SUMI,  SUS  or  a customized 
questionnaire  (priority  3) 


G.3  Product  Details 


Name  and  Version: 


ClFWorks  Web  Software  Delivery  Release  1 


Parts  for  which  requirements  provided:  All  parts  of  the  software 


Intended  user  groups: 


System  Administrator,  experienced  with  server  software  installation 


Training  required: 


None  (for  intended  user  groups) 


Work  supported  by  product: 


Downloading  and  preparing  ClFWorks’  Site  Guard  software  for 
installation. 


Usage  environment: 


Medium  to  large  size  corporate  and  public  sector  companies. 


Users  with  special  needs: 


Product  will  meet  standards  for  Section  508  of  the  Rehabilitation  Act 


33 


NIST/IUSR  - CISU-R 


G.4  Product  Objective 

The  objective  of  the  product  is  to  replace  mailing  DVD’s  as  the  default  mechanism  for  shipping  ClFWorks’ 
software. 


G.5  Context  of  Use 

The  context  of  use  section  of  this  document  has  not  been  changed  from  the  Level  1 version  (15  May  2006) 

G. 5.1.  Users 

G. 5.1.1. System  Administrators 

System  Administrators  are  experienced  computer  professionals  whose  primary  roles  are  the  installation, 
configuration,  and  administration  of  multi-user,  production,  software  . These  users  are  typically  administrators 
of  other  enterprise  software  products  such  as  operating  systems,  databases,  and  web  servers. 

System  Administrator’s  educational  backgrounds  are  varied  as  some  receive  formal  academic  training  in 
computer  science  or  computer  systems  management  while  others  receive  training  on  the  job  or  through 
vendor  or  3rd  party  courses. 

The  most  important  characteristic  of  a System  Administrator  is  on  the  job  experience.  There  are  typically  two 
routes  by  which  these  users  gain  experience  as  a System  Administrator: 

- They  extend  their  background  from  being  a PC  administrator  or  power  user  into  the  role  of  a System 
Administrator  by  industry  training, 

- They  grow  their  skills  in  an  organization  under  the  tutelage  of  a Senior  System  Administrator. 

For  Web  Software  Delivery  we  expect  System  Administrators  to  have  1-10  years  of  experience  in  their  job.  In 
addition  to  software  installation,  and  configuration,  their  day  to  day  tasks  would  include  operations  (adding 
users,  adding  disk  space,  monitoring  an  existing  system),  and  troubleshooting.  They  may  also  perform 
planning  operations  for  future  software  and  hardware  acquisitions. 

G.5. 1.2. Users  with  Disabilities: 

Users  with  vision,  hearing,  and  motor  disabilities  should  be  able  to  function  as  a System  Administrator  with 
appropriate  assistive  devices  (e.g.  screen  readers). 

G.5. 2.  Goals 

The  general  goal  is  to  obtain  the  appropriate  version  of  the  ClFWorks  Site  Guard  software  for  the  appropriate 
platform  in  a form  ready  for  installation.  Obtaining  the  software  completes  the  purchasing  process  but  may  or 
may  not  lead  immediately  to  installation.  In  some  cases  hardware  or  software  has  to  be  configured  before 
installation  of  the  software  begins.  Users  may  immediately  burn  the  electronic  version  of  the  software  onto 
DVD’s,  however  this  is  not  required  for  installation.  Some  users  will  only  install  the  product  from  the 
downloaded  files  (referred  to  as  a “staged  install”).  Based  on  this  there  are  three  tasks  associated  with 
electronic  delivery. 

Task  1.  Finding  and  downloading  the  Site  Guard  software  via  Web  Software  Delivery 
Task  2.  Create  DVD’s  from  the  downloaded  Site  Guard  software  and  begin  the  install. 

Task  3.  Prepare  the  software  for  an  electronic  or  “staged”  install  and  begin  the  install 
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G.5.3.  Equipment 

To  use  Web  Software  Delivery  the  following  computing  environment  will  be  required. 

- A computer  capable  of  running  the  browsers  listed  below. 

- Internet  Explorer  (6.0  and  higher),  Netscape  (6.0  and  higher),  or  Mozilla  Firefox  1.1  and  higher 

- 10  gigabytes  of  disk  space  to  download  the  zip  files  required  for  installation  and  create  the  staged 
installation. 

- 500  megabytes  of  RAM 

- A lOmbs  or  faster  internet  connection. 

- A SVGA  compatible  monitor. 

- A DVD  Burner 

G.5.4.  Physical  and  Social  Environment 

- The  physical  environment  will  be  either  a corporate/  public  sector  office  or  a home  office  with  a high 
speed  internet  connection  (10  mbs  or  faster).  The  home  office  may  be  used  to  download  the  software 
and  burn  CD’s  after  hours. 

- Typically  system  administrators  multi-task.  It  is  expected  that  using  web  software  delivery  will  be 
mixed  in  with  other  support  tasks  they  are  completing. 

- There  is  typically  a time  lag  between  acquiring  and  installing  the  software,  and  rolling  out  the  software 
into  production.  Because  of  that  acquiring  and  installing  the  software  will  likely  be  part  of  a planning 
exercise  without  mission  criticality  pressure.  However  due  to  hacker  attacks  or  other  security 
concerns  there  may  be  times  when  obtaining  Site  Guard  may  be  urgent. 

G.5.5.  Scenarios  of  use 

Scenario  1:  Download  the  Site  Guard  software  and  burn  it  to  DVD 

Ann  is  the  lead  system  administrator  at  Biagra  a producer  of  genetically  engineered  wheat  products.  Last 
month  her  company  was  hit  by  a denial  of  service  attack  by  a group  of  organic  fundamentalists.  It  was  a 
rough  month  and  Ann  ended  up  shutting  down  the  company  website  leading  to  unhappy  customers  and 
employees.  It  took  no  time  to  get  the  PO  approved  for  Site  Guard  and  now  she  is  on  the  hot-seat  to  get  it 
installed  on  all  the  companies  servers.  She  is  glad  she  can  get  the  software  online  since  she  does  not 
want  to  wait  for  a physical  shipment.  Since  her  company's  firewall  is  closed,  she  has  to  download  the 
software  at  home  and  take  it  into  the  office  and  install  it  on  her  2 primary  servers. 

After  dinner  she  turns  on  Toy  Story  2 for  the  kids  and  sits  down  in  the  study  to  download  the  software. 
She  has  the  printed  out  email  from  ClFWorks  and  uses  it  to  login  to  the  Web  Delivery  site.  After  login, 
she  sees  the  software  she  purchased  and  online  instructions  for  how  to  download  it.  She  uses  Mozilla  for 
the  download  as  it  has  a built  in  download  manager.  She  begins  the  first  DVD  download  and  then  goes 
to  check  on  the  kids.  Occasionally  she  peeks  in  on  the  computer  to  see  how  the  download  is  going.  She 
is  happy  to  see  that  it  is  progressing  nicely,  although  it  is  still  a bit  slow.  10  minutes  later  the  first  DVD  is 
downloaded  so  she  starts  the  second  DVD  download.  20  minutes  later  she  has  put  the  kids  to  bed  and 
comes  back  in  to  see  that  the  second  DVD  download  is  complete. 

Going  back  to  the  ClFWorks  website  she  launches  a utility  that  will  help  her  burn  the  DVD's.  It  asks  her  for  her 
computer  information  and  then  asks  her  to  insert  a DVD.  While  the  DVD  is  burning  she  sees  a progress 
indicator.  It  says  20  minutes  to  go,  so  she  surfs  the  internet  for  a while.  Twenty  minutes  later  she  starts  the 
second  DVD.  At  the  end  of  the  evening  she  has  two  DVD's  ready  for  installation  in  the  morning. 
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Back  at  work,  she  takes  the  DVD’s  and  pops  them  into  her  development  environment.  The  development 
environment  is  a clone  of  the  production  web  server  that  she  uses  to  test  out  installations.  She  opens  a 
Unix  Shell  and  iaunches  the  installer.  She  is  glad  to  see  that  the  installation  starts  successfully. 

Scenario  2:  Download  the  Site  Guard  software  and  create  a staged  installation 

Winston  is  a Senior  System  Administrator  for  the  University  of  South-East  North  Dakota.  He  administers 
the  systems  that  run  the  school’s  websites.  He  takes  pride  in  maintaining  a safe  system  despite  the 
many  hackers  at  the  school.  As  part  of  his  2005  budget,  he  got  approval  to  purchase  Site  Guard  and 
deploy  it  to  the  server  farm  that  is  used  for  the  university  website. 

Today  he  received  the  email  from  ClFWorks  with  his  license  number  and  instructions  for  downloading  the 
software.  Winston  always  likes  to  install  software  from  a staged  environment.  This  makes  it  much  faster 
to  take  a master  copy  of  the  software  and  install  it  on  multiple  machines  in  his  server  farm.  He  has 
designated  a system  called  “stage_guard”  to  be  the  stage  for  Site  Guard.  He  prints  out  the  email  (he 
can’t  access  it  from  stage_guard)  and  walks  over  to  the  stage_guard  machine.  He  logs  onto  the 
ClFWorks  website.  He  sees  the  software  that  he  needs  to  download  and  begins  the  first  download. 
While  the  first  download  is  completing  he  looks  around  on  the  website  for  instruction  on  building  a staged 
install.  He  sees  a link  to  instructions  for  staged  install  on  the  side  of  the  web  page.  He  clicks  the  link  and 
starts  reading  the  instructions.  The  instructions  say  he  can  either  do  it  manually  or  download  a script. 
Being  security  conscious  he  does  not  trust  “foreign”  scripts  so  he  does  it  the  manual  way. 

He  follows  the  instructions  and  begins  creating  the  directories  he  will  need  for  the  install.  It  takes  a while 
to  create  all  the  directories,  as  he  has  to  create  them  using  command  line  UNIX  and  then  set  privileges 
for  them.  He  checks  back  on  the  first  download  and  sees  it  is  complete.  He  starts  the  second  download. 
While  that  is  working  he  completes  the  directories  he  needs  and  focuses  on  unzipping  the  files.  The 
instructions  say  to  use  a zip  utility  to  unpackage  and  put  the  files  in  the  proper  directories.  He 
appreciates  the  fact  that  the  instructions  list  the  exact  command  to  use  with  his  zip  utility.  He  unzips  the 
files  from  the  first  download  and  puts  them  into  the  appropriate  directory.  After  that  is  complete  he 
checks  back  in  with  the  download  and  sees  the  second  download  is  complete.  He  unzips  that  file  and 
puts  it  into  the  second  directory.  He  goes  back  to  the  first  directory  and  runs  the  installer.  He  is  glad  to 
see  it  has  come  up  and  he  can  proceed  with  the  installation. 


G.6  Performance  and  Satisfaction  Criteria 

The  performance  and  satisfaction  criteria  have  been  updated  in  this  version  to  add  target  values,  and  to 
further  define  the  criteria  and  requirements  for  testing  them. 

G.  6.1.  Goals 

The  goal  for  using  Web  Software  Delivery  is  to  find  and  download  the  software  for  the  appropriate  platform  in 
a form  ready  for  installation.  This  goal  is  completed  when  the  actual  installation  begins,  or  is  ready  to  begin. 

There  are  three  core  tasks  to  complete  this  goal  with  Web  Software  Delivery. 

1 . Download  SITE  guard  software 

This  task  has  the  highest  priority  since  successful  delivery  of  the  software  is  a prerequisite  to 
the  other  tasks. 

2.  Burn  SITE  guard  software  to  DVD  and  launch  installer 

This  task  has  second  priority  since  it  is  the  recommended  approach  for  installing  the  software. 

3.  Prepare  SITE  guard  software  for  staged  install  and  launch  installer 
This  task  is  lowest  priority  as  it  is  an  optional  way  of  installation. 
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G.6.2.  Criteria 

The  table  below  lists  the  key  tasks  associated  with  Web  Software  Delivery  and  the  usability  requirements  for 
effectiveness,  efficiency,  and  satisfaction.  These  target  values  were  determined  through  interviews  and 
observations  of  typical  users  completing  comparable  tasks,  in  consultation  with  the  deployment  manager  and 
marketing.  They  will  be  confirmed  or  updated  with  benchmark  usability  testing  of  the  current  delivery  methods 
to  ensure  that  these  target  criteria  would  result  in  an  improvement  over  the  current  delivery  methods. 

- Effectiveness:  the  unassisted  task  completion  rate. 

Unassisted  task  completion  rate  is  the  percentage  of  users  who  complete  the  tasks  without  assists 
from  the  usability  test  administrator.  This  criteria  is  the  first  priority,  as  successful  completion  of  each 
task  is  the  most  important  requirement  for  users. 

- Efficiency:  the  average  time  to  complete  the  task 

Efficiency  will  be  measured  as  the  average  number  of  minutes  to  complete  each  task.  Only  user  time 
will  be  measured.  Machine  time  (for  example,  actual  download  time)  will  not  be  counted.  This  is  the 
second  priority,  as  the  time  taken  by  the  machine  is  expected  to  be  longer  than  the  user  time. 

- Satisfaction,  using  the  SUMI  questionnaire,  administered  after  completing  the  three  tasks.  This  is  also 
a second  priority  because  any  lack  of  satisfaction  will  reduce  users’  willingness  to  continue  to  use  web 
delivery. 


Task 

Effectiveness: 

Unassisted  task 
completion  rate  of  at 
least 

Efficiency: 

Maximum  user  time 
(not  machine  time)  of 

Satisfaction: 

Post-task  SUMI  score 
of  at  least: 

Download  SITE  guard 
software 

90% 

10  minutes 

Burn  SITE  quard  software  to 
DVD 

80% 

15  minutes 

Prepare  SITE  guard 
software  for  staged  install 
and  launch  installer 

70% 

30  minutes 

Product  overall 

50 

Table  1:  Target  criteria  values 


G.7  Methods  for  Testing 

The  product  will  be  evaluated  against  the  performance  and  satisfaction  criteria  with  a usability  test,  using  test 
scenarios  based  on  the  three  scenarios. 

G.7.1.  Users 

A minimum  of  8 users  should  test  the  product.  All  users  should  be  System  Administrators  with  at  least  1 year 
of  experience  as  a system  administrator  managing  server  software. 

G.7. 2.  Goals 

Refer  to  Performance  and  Satisfaction  Criteria  in  the  body  of  the  document  for  Tasks  and  Goals. 

G.7. 3.  Test  facility 

The  test  should  be  conducted  in  a usability  lab  that  consists  of  a user  room  and  a control  room.  The  user  room 
will  be  an  office  where  the  participants  conduct  the  tasks.  The  lab  should  be  equipped  with  audio  and  video 
monitoring  and  recording  devices,  as  well  as  a scan  converter  that  allows  the  display  on  the  participant’s 
monitor  to  be  video  recorded. 
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G.7.4.  Participant’s  computing  environment 

The  participants  computing  environment  should  meet  the  minimum  set  forth  in  the  section  “Equipment”  in  the 
main  body  of  this  document. 

G.7.5.  Display  devices 

As  mentioned  in  the  “equipment”  section  of  this  document,  a minimum  of  a color  VGA  monitor  with  capability 
of  800x600  resolution  is  required. 

G.7.6.  Experimental  Design 

Tasks  will  be  presented  to  each  participant  in  the  same  order  because  since  the  second  and  third  tasks  are 
dependent  on  the  first  one.  The  six  dependent  measures  for  this  study  should  be: 

- Unassisted  task  completion  rate  (the  percentage  of  participants  who  completed  a task  without 
assistance  from  the  tester) 

- Assisted  task  completion  rate  (the  cumulative  percentage  of  people  who  completed  a task  with  zero, 
one  or  two  assists  from  the  tester) 

- Task  time 

- Number  of  errors 

- Number  of  assists 

- Self-report  satisfaction  via  SUMI  measures. 
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Annex  H (Informative) 

An  example  of  a Level  3 requirements  document 


H.1  Title  Page 

Usability  Requirements 

for  Web  Software  Delivery  Release  1 

ClFWorks 


18  November  2006 

Requirements  produced  by:  Sherman  Jones. 

Contributors:  Fred  Smith,  Product  Manager 

Reviewers:  Ann  Smith,  VP  of  Ul 

Alexander  Kushenko,  Product  Boss 
James  Dearing,  Director  of  Marketing 
Jennifer  Taylor,  Deployment  Manager 
John  Smith,  User  Researcher 
Jackie  Mayer,  Usability  Lab  Manager 


Created  under  the  Common  Industry  Specification  for  Usability  - Requirements  (CISU-R)  v0.88 


Address  inquiries  to: 
Phone: 

Email: 

Address: 


Sherman  Jones,  ClFWorks 
650-555-1212 

sherman.jones@CIFWorks.com 

150  New  Parkway,  New  Pines,  CA  9101 1 


Version  History 


10  May  2006 

1.0 

Sherman  Jones 

Created  document 

15  May  2006 

1.1 

Sherman  Jones 

Revised  based  on  review  (Level  1 complete) 

10  July  2006 

2.0 

Sherman  Jones 

Added  detail  for  criteria 

18  Aug  2006 

2.1 

John  Smith 

Added  preliminary  test  method 

25  Aug  2006 

2.2 

Sherman  Jones 

Added  target  values  (Level  2 complete) 

15  Oct  2006 

3.1 

Sherman  Jones 

Adjusted  target  values  to  benchmark  test 

18  Nov  2006 

3.2 

John  Smith 

Finalized  testing  protocol  (Level  3 complete) 

Related  Documents 

"New  Product  Initiation  Plan”  - 13  April  2006  - Alexander  Kushenko 
"Product  competitive  landscape  review”  - 8 July  2006  - James  Dearing 
"Site  visits  and  user  observations  report”  - 5 August  2006  - John  Smith 
"Product  specifications  document"  - 10  August  2006  - Fred  Smith 
"User  interface  specifications”  - 15  August  2006  - Ann  Smith 
"Benchmarking  usability  test  report"  - 3 October  2006  - Jackie  Mayer 


H.2  Executive  Summary 

The  purpose  of  this  document  is  to  specify  the  usability  requirements  for  the  first  release  of  Web  Software 
Delivery  in  a format  that  allows  ClFWorks’  customers  to  review  and  provide  input. 
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This  document  conforms  to  Level  3 of  the  CISU-R.  It  updates  the  previous  Level  3 document  with: 

- Updated  target  values  for  the  performance  and  satisfaction  criteria,  as  confirmed  through  benchmark 
usability  testing 

- Complete  usability  testing  protocol  to  test  the  product  against  the  criteria 

The  objective  of  the  Web  Software  Delivery  product  is  to  move  away  from  physical  shipments  of  CD’s  and 
DVD’s  towards  web  based  delivery  of  ClFWorks’  products.  The  key  tasks  involved  are  downloading  the 
software,  creating  DVDs  from  the  software  and  setting  up  a staged  (does  not  use  DVDs)  installation  from  the 
downloaded  files. 

The  intended  users  of  Web  Software  Delivery  are  system  administrators.  System  administrators  are 
experienced  computer  professionals  whose  primary  roles  are  the  installation,  configuration,  and  administration 
of  multi-user,  production  software  packages.  They  will  use  the  product  in  a corporate  computing  environment 
on  a computer  with  a recent  version  of  the  browser,  sufficient  disk  space  to  download  the  product  and  a high 
speed  (lOmbs  or  faster)  internet  connection. 

The  key  tasks,  for  which  requirements  scenarios  have  been  developed  are: 

- Download  SITE  guard  software 

- Burn  SITE  guard  software  to  DVD 

- Prepare  SITE  guard  software  for  staged  install  and  launch  installer 
The  usability  of  this  product  will  be  tested  against  the  following  criteria: 

- Effectiveness:  the  unassisted  task  completion  rate,  measured  as  a percentage  of  total  task  attempts 
(priority  1 ) 

- Efficiency:  the  average  time  to  complete  the  task,  measured  as  the  average  number  of  minutes  to 
complete  each  task  (priority  2) 

- Satisfaction,  using  the  SUMI  questionnaire  (priority  3) 

The  table  below  lists  the  key  tasks  associated  with  Web  Software  Delivery  and  the  usability  requirements  for 
effectiveness,  efficiency,  and  satisfaction.  A usability  test  will  be  conducted  on  the  released  product  to 
measure  if  the  requirements  criteria  have  been  met. 


Task 

Effectiveness: 

Unassisted  task  completion 
rate  of  at  least 

Efficiency: 

Maximum  user  time  (not 
machine  time)  of 

Satisfaction: 

Post-task  SUMI  score 
of  at  least: 

Download  SITE  guard 
software 

90  % 

10  minutes 

Burn  SITE  quard  software  to 
DVD 

90  % 

15  minutes 

Prepare  SITE  guard 
software  for  staged  install 
and  launch  installer 

75  % 

25  minutes 

Product  overall  scores 

75 
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H.3  Product  Details 


Name  and  Version: 


ClFWorks  Web  Software  Delivery  Release  1 


Parts  for  which  requirements  provided:  All  parts  of  the  software 


Intended  user  groups: 


System  Administrator,  experienced  with  server  software  installation 


Training  required: 


None  (for  intended  user  groups) 


Work  supported  by  product: 


Downloading  and  preparing  ClFWorks’  Site  Guard  software  for 
installation. 


Usage  environment: 


Medium  to  large  size  corporate  and  public  sector  companies. 


Users  with  special  needs: 


Product  will  meet  standards  for  Section  508  of  the  Rehabilitation  Act 


H.4  Product  Objective 

The  objective  of  the  product  is  to  ultimately  replace  mailing  DVD's  as  the  default  mechanism  for  shipping 
ClFWorks’  software. 


H.5  Context  of  Use 

The  context  of  use  section  of  this  document  has  not  been  changed  from  the  Level  1 version  (15  May  2006) 

H.5.1.  Users 
System  Administrators 

System  Administrators  are  experienced  computer  professionals  whose  primary  roles  are  the  installation, 
configuration,  and  administration  of  multi-user,  production,  software  . These  users  are  typically  administrators 
of  other  enterprise  software  products  such  as  operating  systems,  databases,  and  web  servers. 

System  Administrator’s  educational  backgrounds  are  varied  as  some  receive  formal  academic  training  in 
computer  science  or  computer  systems  management  while  others  receive  training  on  the  job  or  through 
vendor  or  3rd  party  courses. 

The  most  important  characteristic  of  a System  Administrator  is  on  the  job  experience.  There  are  typically  two 
routes  by  which  these  users  gain  experience  as  a System  Administrator: 

- They  extend  their  background  from  being  a PC  administrator  or  power  user  into  the  role  of  a System 
Administrator  by  industry  training, 

- They  grow  their  skills  in  an  organization  under  the  tutelage  of  a Senior  System  Administrator. 

For  Web  Software  Delivery  we  expect  System  Administrators  to  have  1 -1 0 years  of  experience  in  their  job.  In 
addition  to  software  installation,  and  configuration,  their  day  to  day  tasks  would  include  operations  (adding 
users,  adding  disk  space,  monitoring  an  existing  system),  and  troubleshooting.  They  may  also  perform 
planning  operations  for  future  software  and  hardware  acquisitions. 

H. 5.1.1.  Users  with  Disabilities: 

Users  with  vision,  hearing,  and  motor  disabilities  should  be  able  to  function  as  a System  Administrator  with 
appropriate  assistive  devices  (e.g.  screen  readers). 
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H. 6  Goals 

The  general  goal  is  to  obtain  the  appropriate  version  of  the  ClFWorks  Site  Guard  software  for  the  appropriate 
platform  in  a form  ready  for  installation.  Obtaining  the  software  completes  the  purchasing  process  but  may  or 
may  not  lead  immediately  to  installation.  In  some  cases  hardware  or  software  has  to  be  configured  before 
installation  of  the  software  begins.  Users  may  immediately  burn  the  electronic  version  of  the  software  onto 

DVD's,  however  this  is  not  required  for  installation.  Some  users  will  only  install  the  product  from  the 

downloaded  files  (referred  to  as  a “staged  install").  Based  on  this  there  are  three  tasks  associated  with 
electronic  delivery. 

I.  Finding  and  downloading  the  Site  Guard  software  via  Web  Software  Delivery 

2.  Create  DVD’s  from  the  downloaded  Site  Guard  software  and  begin  the  install. 

3.  Prepare  the  software  for  an  electronic  or  “staged”  install  and  begin  the  install 


In  the  case  of  tasks  2 and  3,  the  tasks  end  when  the  user  has  started  the  installation  successfully. 

H.7  Equipment 

To  use  Web  Software  Delivery  the  following  computing  environment  will  be  required. 

- A computer  capable  of  running  the  browsers  listed  below. 

- Internet  Explorer  (6.0  and  higher),  Netscape  (6.0  and  higher),  or  Mozilla  Firefox  1.1  and  higher 

- 10  gigabytes  of  disk  space  to  download  the  zip  files  required  for  installation  and  create  the  staged 
installation. 

- 500  megabytes  of  RAM 

- A lOmbs  or  faster  internet  connection. 

- A SVGA  compatible  monitor. 

- A DVD  Burner 


H.8  Physical  and  Social  Environment 

- The  physical  environment  will  be  either  a corporate/  public  sector  office  or  a home  office  with  a high 
speed  internet  connection  (10  mbs  or  faster).  The  home  office  may  be  used  to  download  the  software 
and  burn  CD’s  after  hours. 

- Typically  system  administrators  multi-task.  It  is  expected  that  using  web  software  delivery  will  be 
mixed  in  with  other  support  tasks  they  are  completing. 

- There  is  typically  a time  lag  between  acquiring  and  installing  the  software,  and  rolling  out  the  software 
into  production.  Because  of  that  acquiring  and  installing  the  software  will  likely  be  part  of  a planning 
exercise  without  mission  criticality  pressure.  However  due  to  hacker  attacks  or  other  security 
concerns  there  may  be  times  when  obtaining  Site  Guard  may  be  urgent. 


H.9  Scenarios  of  use 

Scenario  1:  Download  the  Site  Guard  software  and  burn  it  to  DVD 
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Ann  is  the  lead  system  administrator  at  Biagra  a producer  of  genetically  engineered  wheat  products.  Last 
month  her  company  was  hit  by  a denial  of  service  attack  by  a group  of  organic  fundamentalists.  It  was  a 
rough  month  and  Ann  ended  up  shutting  down  the  company  website  leading  to  unhappy  customers  and 
employees.  It  took  no  time  to  get  the  PO  approved  for  Site  Guard  and  now  she  is  on  the  hot-seat  to  get  it 
installed  on  all  the  companies  servers.  She  is  glad  she  can  get  the  software  online  since  she  does  not 
want  to  wait  for  a physical  shipment.  Since  her  company’s  firewall  is  closed,  she  has  to  download  the 
software  at  home  and  take  it  into  the  office  and  install  it  on  her  2 primary  servers. 

After  dinner  she  turns  on  Toy  Story  2 for  the  kids  and  sits  down  in  the  study  to  download  the  software. 
She  has  the  printed  out  email  from  ClFWorks  and  uses  it  to  login  to  the  Web  Delivery  site.  After  login, 
she  sees  the  software  she  purchased  and  online  instructions  for  how  to  download  it.  She  uses  Mozilla  for 
the  download  as  it  has  a built  in  download  manager.  She  begins  the  first  DVD  download  and  then  goes 
to  check  on  the  kids.  Occasionally  she  peeks  in  on  the  computer  to  see  how  the  download  is  going.  She 
is  happy  to  see  that  it  is  progressing  nicely,  although  it  is  still  a bit  slow.  10  minutes  later  the  first  DVD  is 
downloaded  so  she  starts  the  second  DVD  download.  20  minutes  later  she  has  put  the  kids  to  bed  and 
comes  back  in  to  see  that  the  second  DVD  download  is  complete. 

Going  back  to  the  ClFWorks  website  she  launches  a utility  that  will  help  her  burn  the  DVD's.  It  asks  her 
for  her  computer  information  and  then  asks  her  to  insert  a DVD.  While  the  DVD  is  burning  she  sees  a 
progress  indicator.  It  says  20  minutes  to  go,  so  she  surfs  the  internet  for  a while.  Twenty  minutes  later 
she  starts  the  second  DVD.  At  the  end  of  the  evening  she  has  two  DVD's  ready  for  installation  in  the 
morning. 

Back  at  work,  she  takes  the  DVD’s  and  pops  them  into  her  development  environment.  The  development 
environment  is  a clone  of  the  production  web  server  that  she  uses  to  test  out  installations.  She  opens  a 
Unix  Shell  and  launches  the  installer.  She  is  glad  to  see  that  the  installation  starts  successfully. 

Scenario  2:  Download  the  Site  Guard  software  and  create  a staged  installation 

Winston  is  a Senior  System  Administrator  for  the  University  of  South-East  North  Dakota.  He  administers 
the  systems  that  run  the  school’s  websites.  He  takes  pride  in  maintaining  a safe  system  despite  the 
many  hackers  at  the  school.  As  part  of  his  2005  budget,  he  got  approval  to  purchase  Site  Guard  and 
deploy  it  to  the  server  farm  that  is  used  for  the  university  website. 

Today  he  received  the  email  from  ClFWorks  with  his  license  number  and  instructions  for  downloading  the 
software.  Winston  always  likes  to  install  software  from  a staged  environment.  This  makes  it  much  faster 
to  take  a master  copy  of  the  software  and  install  it  on  multiple  machines  in  his  server  farm.  He  has 
designated  a system  called  “stage_guard”  to  be  the  stage  for  Site  Guard.  He  prints  out  the  email  (he 
can’t  access  it  from  stage_guard)  and  walks  over  to  the  stage_guard  machine.  He  logs  onto  the 
ClFWorks  website.  He  sees  the  software  that  he  needs  to  download  and  begins  the  first  download. 
While  the  first  download  is  completing  he  looks  around  on  the  website  for  instruction  on  building  a staged 
install.  He  sees  a link  to  instructions  for  staged  install  on  the  side  of  the  web  page.  He  clicks  the  link  and 
starts  reading  the  instructions.  The  instructions  say  he  can  either  do  it  manually  or  download  a script. 
Being  security  conscious  he  does  not  trust  “foreign"  scripts  so  he  does  it  the  manual  way. 

He  follows  the  instructions  and  begins  creating  the  directories  he  will  need  for  the  install.  It  takes  a while 
to  create  all  the  directories,  as  he  has  to  create  them  using  command  line  UNIX  and  then  set  privileges 
for  them.  He  checks  back  on  the  first  download  and  sees  it  is  complete.  He  starts  the  second  download. 
While  that  is  working  he  completes  the  directories  he  needs  and  focuses  on  unzipping  the  files.  The 
instructions  say  to  use  a zip  utility  to  unpackage  and  put  the  files  in  the  proper  directories.  He 
appreciates  the  fact  that  the  instructions  list  the  exact  command  to  use  with  his  zip  utility.  He  unzips  the 
files  from  the  first  download  and  puts  them  into  the  appropriate  directory.  After  that  is  complete  he 
checks  back  in  with  the  download  and  sees  the  second  download  is  complete.  He  unzips  that  file  and 
puts  it  into  the  second  directory.  He  goes  back  to  the  first  directory  and  runs  the  installer.  He  is  glad  to 
see  it  has  come  up  and  he  can  proceed  with  the  installation. 
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H.10  Performance  and  Satisfaction  Criteria 

The  performance  and  satisfaction  criteria  have  been  updated  in  this  version  to  update  target  values,  based  on 
results  of  benchmark  usability  testing  and  additional  user  research  via  web  surveys. 


H.10.1.  Goals 

The  goal  for  using  Web  Software  Delivery  is  to  find  and  download  the  software  for  the  appropriate  platform  in 
a form  ready  for  installation.  This  goal  is  completed  when  the  actual  installation  begins,  or  is  ready  to  begin. 

There  are  three  core  tasks  to  complete  this  goal  with  Web  Software  Delivery. 

1 . Download  SITE  guard  software 

This  task  has  the  highest  priority  since  successful  delivery  of  the  software  is  a prerequisite  to 
the  other  tasks. 

2.  Burn  SITE  guard  software  to  DVD  and  launch  installer 

This  task  has  second  priority  since  it  is  the  recommended  approach  for  installing  the  software. 

3.  Prepare  SITE  guard  software  for  staged  install  and  launch  installer 
This  task  is  lowest  priority  as  it  is  an  optional  way  of  installation. 

H. 10.2. Criteria 

The  table  below  lists  the  key  tasks  associated  with  Web  Software  Delivery  and  the  usability  requirements  for 
effectiveness,  efficiency,  and  satisfaction.  These  target  values  were  determined  through  interviews  and 
observations  of  typical  users  completing  comparable  tasks,  in  consultation  with  the  deployment  manager  and 
marketing,  and  updated  based  on  benchmark  usability  testing  of  the  current  delivery  methods  to  ensure  that 
these  target  criteria  would  result  in  an  improvement  over  the  current  delivery  methods. 

- Effectiveness:  the  unassisted  task  completion  rate. 

Unassisted  task  completion  rate  is  the  percentage  of  users  who  complete  the  tasks  without  assists 
from  the  usability  test  administrator.  This  criteria  is  the  first  priority,  as  successful  completion  of  each 
task  is  the  most  important  requirement  for  users. 

- Efficiency:  the  average  time  to  complete  the  task 

Efficiency  will  be  measured  as  the  average  number  of  minutes  to  complete  each  task.  Only  user  time 
will  be  measured.  Machine  time  (for  example,  actual  download  time)  will  not  be  counted.  This  is  the 
second  priority,  as  the  time  taken  by  the  machine  is  expected  to  be  longer  than  the  user  time. 

Satisfaction,  using  the  SUMI  questionnaire,  administered  after  completing  the  three  tasks.  This  is  also 
a second  priority  because  any  lack  of  satisfaction  will  reduce  users'  willingness  to  continue  to  use  web 
delivery. 

The  scenarios  of  usage  for  each  task  are  listed  below  with  the  target  values  for  each  task 


Task 

Effectiveness: 

Unassisted  task 
completion  rate  of  at 
least 

Efficiency: 

Maximum  user  time 
(not  machine  time)  of 

Satisfaction: 

Post-task  SUMI  score 
of  at  least: 

Download  SITE  guard 
software 

90  % 

10  minutes 

Burn  SITE  guard  software  to 
DVD 

80  % 

15  minutes 

Prepare  SITE  guard 
software  for  staged  install 
and  launch  installer 

70  % 

30  minutes 

Product  overall 

50 
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Table  1:  Target  criteria  values 

The  unassisted  task  completion  rate  was  set  at  90  % since  it  represents  the  highest  unassisted  task 
completion  rate  that  could  realistically  be  achieved  given  user  variability  in  a particular  usability  test. 

The  unassisted  task  completion  rate  for  Task  3 (staged  install)  was  set  to  70  %.  Building  a staged  install  is  a 
more  complex  approach  to  Web  Software  Delivery  that  involves  manual  steps.  ClFWorks  will  support  this 
approach  with  instructions  but  given  the  variability  in  user  environments  and  skill  we  expect  the  success  rate 
to  be  lower.  We  consider  this  acceptable  since  there  will  be  a smaller  percentage  of  users  who  use  staged 
install  than  DVD  install.  In  addition  we  will  have  a help  desk  available  to  walk  users  through  this  approach. 

The  target  value  for  efficiency  (task  completion  time)  was  set  for  all  tasks  based  on  a marketing  research 
survey  done  on  Web  software  delivery,  and  confirmed  through  benchmark  usability  testing.  All  times  listed 
are  for  user  interaction  only  (not  machine  time).  Given  the  high  variability  of  networks  and  machines  we  could 
not  accurately  predict  machine  time  for  these  tasks. 

The  target  value  for  Satisfaction  was  based  on  the  SUMI  questionnaire.  SUMI  is  an  industry  standard  tool  for 
measuring  user  satisfaction.  The  SUMI  score  will  be  calculated  by  aggregating  the  post  test  survey  of  each 
individual  participant.  The  industry  average  for  SUMI  questionnaires  is  50,  but  given  the  importance  of  the 
business  goal  of  converting  users  to  Web  delivery,  the  value  of  75  was  chosen  as  ClFWorks’  benchmark  for 
user  satisfaction  as  measured  by  SUMI. 

For  all  tasks  the  user  is  a System  Administrator  (as  defined  in  Context  of  Use:  Users)  with  between  1-10  years 
of  experience  administering  server  based  software.  Tasks  1 and  2 may  be  completed  in  either  a home  office 
with  a high  speed  internet  connection  or  in  an  office  setting.  Task  3 would  be  completed  in  an  office  setting. 

For  tasks  1 and  2 the  effectiveness  requirements  are  definitive  (not  subject  to  negotiation)  due  to  corporate 
directive  at  ClFWorks.  The  efficiency  and  satisfaction  metrics  for  Tasks  1 and  2 are  provisional  requirements, 
which  are  open  to  negotiation  with  customers  of  the  software.  All  requirements  for  Task  3 are  objectives  for 
guidance  intended  for  the  internal  development  team.  We  cannot  predict  or  control  the  breadth  of 
environments  at  customers’  sites  so  we  are  keeping  the  requirements  at  a high  level. 


H.11  Annex  A:  Usability  test  requirements  format 

This  annex  provides  a format  for  specifying  the  method  to  be  used  to  test  whether  the  usability  requirements 
have  been  met.  The  format  is  similar  to  CIF  (ISO/IEC  25062),  and  the  results  of  the  test  can  subsequently  be 
documented  using  the  CIF. 

H. 11.1. Users 

A minimum  of  8 users  should  test  the  product.  All  users  should  be  System  Administrators  with  at  least  1 year 
of  experience  as  a system  administrator  managing  server  software. 

H. 11.2. Goals 

Refer  to  Performance  and  Satisfaction  Criteria  in  the  body  of  the  document  for  Tasks  and  Goals. 


H.11. 3. Test  facility 

The  test  should  be  conducted  in  a usability  lab  that  consists  of  a user  room  and  a control  room.  The  user  room 
will  be  an  office  where  the  participants  conduct  the  tasks.  The  lab  should  be  equipped  with  audio  and  video 
monitoring  and  recording  devices,  as  well  as  a scan  converter  that  allows  the  display  on  the  participant's 
monitor  to  be  video  recorded. 
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H.  11.4. Participant’s  computing  environment 

The  participants  computing  environment  should  meet  the  minimum  set  forth  in  the  section  “Equipment”  in  the 
main  body  of  this  document. 

H.1 1.5. Display  devices 

As  mentioned  in  the  “equipment”  section  of  this  document,  a minimum  of  a color  VGA  monitor  with  capability 
of  800x600  resolution  is  required. 

H.  11. 6. Experimental  Design 

There  will  be  one  order  that  the  tasks  are  presented  in  since  the  second  and  third  tasks  are  dependent  on  the 
first  one.  The  six  dependent  measures  for  this  study  should  be: 

- Unassisted  task  completion  rate  (the  percentage  of  participants  who  completed  a task  without 
assistance  from  the  tester) 

- Assisted  task  completion  rate  (the  cumulative  percentage  of  people  who  completed  a task  with  zero, 
one  or  two  assists  from  the  tester) 

- Task  time 

- Number  of  errors 

- Number  of  assists 

- Self-report  satisfaction  via  SUMI  measures. 


H.1 1.7. Procedure 

A minimum  of  eight  sessions,  one  for  each  participant,  shall  be  conducted.  The  participant  will  work  in  the 
user  room,  while  the  test  administrator  will  observe  from  the  control  room  behind  a one-way  mirror.  A system 
of  microphones  and  speakers  will  allow  the  administrator  and  user  to  communicate.  At  the  beginning  of  each 
session,  participants  will  be  given  a consent  form  and  non-disclosure  agreement.  After  completing  the  forms, 
participants  will  be  instructed  on  the  nature  of  the  testing  session  and  the  software. 

Data  collection  will  begin  as  the  participant  works  through  the  tasks.  As  participants  work,  a time-stamped 
datalog  will  be  created.  The  datalog  will  note  errors,  assists,  user  comments,  and  other  observations  of 
interest  such  as  navigation  strategies.  Upon  completion  of  the  tasks,  participants  will  fill  out  the  SUMI. 
Participants  will  be  thanked  for  their  time,  and  given  their  compensation.  Compensation  for  this  session 
should  be  approximately  $150  given  the  experience  of  the  participant  and  the  length  of  the  study  (2-3  hours). 

H.1 1.8. Participant  general  instructions 

Participants  should  be  told  that  the  study  session  consists  of  two  parts:  working  on  tasks  with  the  application, 
and  filling  out  some  post-task  surveys.  Participants  shall  be  instructed  to  read  each  task  out  loud,  and  to  work 
through  the  tasks,  “as  efficiently  as  possible.”  Participants  shall  be  assured  that  it  is  the  software,  not  them, 
that  is  being  tested,  and  to  feel  free  to  be  honest  in  their  feedback  of  the  software. 

H.1 1.8.1.  Participant  task  instructions 


************************************************************************************** 

Task  Scenarios  - Web  Software  Delivery 


You  (Sandy  Lee)  are  a System  Administrator  working  at  a company  called  Marsh  International  that 
manufactures  fencing  products.  Your  company  is  located  in  Fremont,  California. 
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The  Director  of  Information  Technology  (Dilip)  has  just  signed  a deal  with  ClFWorks  to  use  Site  Guard  to 
protect  your  corporate  internet  site.  The  deal  also  includes  support  and  upgrades. 

Dilip  emails  you  the  ordering  document  that  details  the  contract  with  ClFWorks.  As  part  of  the  deal  with 
ClFWorks,  he  tells  you  that  the  software  will  be  delivered  electronically  via  a download. 

Exercise  1:  Download  the  software 

Using  the  information  in  the  email  open  on  your  desktop,  download  the  database  software  that  your  company 
has  purchased.  Keep  in  mind  the  following. 

1 . You  want  to  download  the  product  for  Solaris,  Spark  64. 

2.  You  want  the  latest  version  of  the  product. 

3.  You  only  want  to  download  the  software  that  you  have  licensed.  You  don’t  want  any  extras  for  trial 
purposes. 

4.  Here  is  your  company  information 

Marsh  International 
500  Woodsy  Lane 
Fremont,  CA  96009 

5.  Here  is  your  information 
Sandy  Lee 

Senior  System  Administrator 
sandy.lee@marsh.com 


Exercise  2:  Make  DVDs 

Now  that  you  have  downloaded  the  software  you  decide  to  make  a set  of  DVDs  of  the  software  for  safe 
keeping.  Using  the  burnable  DVDs  we  have  provided  to  you  create  a set  of  DVDs  that  will  allow  you  to  install 
the  software  from  these  DVDs. 

Exercise  3:  Install  the  database  onto  a Sun  Machine  without  using  the  DVDs 

You  have  a development  machine  you  want  to  install  Site  Guard  onto  (dev2).  However,  the  machine  does  not 
have  a DVD  drive.  Install  the  software  that  you  have  downloaded  onto  the  Sun  Machine  without  using  the 
DVDs  you  have  just  created. 

Tester  Note:  The  task  here  ends  when  the  user  successfully  starts  the  installer  assuming  they  have  completed 
all  other  steps  correctly.  They  don’t  have  to  run  through  the  install  process,  as  we  are  not  testing  the  installer. 


H.12  Usability  metrics 

H.  12.1.  Effectiveness 

Completion  Rate:  Both  assisted  and  unassisted  task  completion  rates  should  be  calculated.  Unassisted 
completion  rate  is  defined  as  the  percentage  of  participants  who  successfully  complete  each  task  without  any 
assistance  from  the  tester.  Assisted  task  completion  rate  is  defined  as  the  percentage  of  participants  who 
complete  each  task  with  two  or  fewer  assists  from  the  tester  (note  that  the  assisted  task  completion  rate 
includes  the  unassisted  task  completion  rate).  Participants  fail  to  complete  tasks  with  assistance  if  they  require 
3 or  more  assists,  exceed  the  maximum  time  limit  for  each  task,  or  if  they  give  up  on  the  task. 
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Errors:  A deviation  from  the  prescribed  method  to  complete  a task  that  did  not  result  in  goal  attainment  shall 
be  counted  as  an  error.  Consulting  the  on-line  help  shall  not  be  recorded  as  an  error. 

Assists:  Any  intervention  by  the  tester  for  any  of  the  following  reasons  shall  be  counted  as  an  assist. 

- Preventing  the  test  participant  from  performing  an  action  that  would  result  in  a condition  from  which 
the  task  could  not  be  completed. 

- Providing  instruction  in  order  for  the  test  participant  to  complete  the  task  or  end  the  task  as  a result  of 
surpassing  the  amount  of  time  allocated  to  complete  the  task  (more  than  4 times  the  benchmark 
time). 

H. 12.2. Efficiency 

Efficiency  relates  the  level  of  effectiveness  achieved  to  the  quantity  of  resources  expended.  Efficiency  is 
generally  assessed  by  the  mean  time  taken  to  achieve  the  task.  Efficiency  may  also  relate  to  other  resources 
(e.g.  total  cost  of  usage).  A common  measure  of  efficiency  is  time  on  task.  For  this  study  efficiency  shall  be 
measured  as  the  average  amount  of  time  required  to  complete  the  tasks. 

H. 12. 3. Satisfaction 

Satisfaction  describes  a user’s  subjective  response  when  using  the  product.  User  satisfaction  may  be  an 
important  correlate  of  motivation  to  use  a product  and  may  affect  performance  in  some  cases.  Questionnaires 
to  measure  satisfaction  and  associated  attitudes  are  commonly  built  using  Likert  and  semantic  differential 
scales. 

SUMI  scores:  SUMI  is  an  instrument  specifically  designed  to  measure  user  satisfaction  with  commercial 
software.  It  has  been  used  with  hundreds  of  thousands  of  participants  and  enables  comparison  with  industry 
norms  for  satisfaction  with  usability  on  6 dimensions:  global  (overall,  efficiency,  affect,  helpfulness,  control, 
and  learning.  An  industry  average  SUMI  score  on  any  index  is  50. 
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